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Abstract 

We propose a novel application of the Simultaneous Orthogonal Match- 
ing Pursuit (S-OMP) procedure for sparsistant variable selection in ultra- 
high dimensional multi-task regression problems. Screening of variables, 
as introduced in Fan and L v f2008), is an efficient and highly scalable way 
to remove many irrelevant variables from the set of all variables, while re- 
taining all the relevant variables. S-OMP can be applied to problems with 
hundreds of thousands of variables and once the number of variables is re- 
duced to a manageable size, a more computationally demanding procedure 
can be used to identify the relevant variables for each of the regression 
outputs. To our knowledge, this is the first attempt to utilize relatedness 
of multiple outputs to perform fast screening of relevant variables. As 
our main theoretical contribution, we prove that, asymptotically, S-OMP 
is guaranteed to reduce an ultra-high number of variables to below the 
sample size without losing true relevant variables. We also provide for- 
mal evidence that a modified Bayesian information criterion (BIG) can 
be used to efficiently determine the number of iterations in S-OMP. We 
further provide empirical evidence on the benefit of variable selection us- 
ing multiple regression outputs jointly, as opposed to performing variable 
selection for each output separately. The finite sample performance of 
S-OMP is demonstrated on extensive simulation studies, and on a genetic 
association mapping problem. 

Keywords; Adaptive Lasso; Greedy forward regression; Orthogonal 
matching pursuit; Multi-output regression; Multi-task learning; Simulta- 
neous orthogonal matching pursuit; Sure screening; Variable selection 



1 Introduction 

Multiple output regression, also known as multi-task regression, with ultra-high 
dimensional inputs commonly arise in problems such as genome-wide associa- 
tion (GWA) mapping in genetics, or stock portfolio prediction in finance. For 
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example, in a GWA mapping problem, the goal is to find a small set of relevant 
single-nucleotide polymorphisms (SNP) (covariates, or inputs) that account for 
variations of a large number of gene expression or clinical traits {responses, 
or outputs) , through a response function that is often modeled via a regression. 
However, this is a very challenging problem for current statistical methods since 
the number of input variables is likely to reach millions, prohibiting even usage 
of scalable implementation of Lasso-like procedures for model selection, which 
are a convex relaxation of a combinatorial subset selection search. Furthermore, 
the outputs in a typical multi-task regression problem are not independent of 
each other, therefore the discovery of truly relevant inputs has to take into con- 
sideration of potential joint effects induced by coupled responses. To appreciate 
this better, consider again the GWA example. Typically, genes in a biological 
pathway are co-expressed as a module and it is often assumed that a causal SNP 
affects multiple genes in one pathway, but not all of the genes in the pathway. In 
order to effectively reduce the dimensionality of the problem and to detect the 
causal SNPs, it is very important to look at how SNPs affect all genes in a bi- 
ological pathway. Since the experimentally collected data is usually very noisy, 
regressing genes individually onto SNPs may not be sufficient to identify the 
relevant SNPs that are only weakly marginally correlated with each individual 
gene in a module. However, once the whole biological pathway is examined, it 
is much easier to find such causal SNPs. In this paper, we demonstrate tha t the 
Simultaneous Orthogonal Matching Pursuit (S-OMP) (jTropp et al.. can 
be used to quickly reduce the dimensionality of such problems, without losing 
any of the relevant variables. 

From a computational point of view, as the dimensionality of the problem 
and the number of outputs increase, it can become intractable to solve the un- 
derlying convex programs commonly used t o identify relevant variables in multi- 
tas k regression proble ms. Previous work bv lLiu et al. iLounici et al. C2009I) 



an dH im et al. f2009t ). for example, do not scale well to settings when the num- 
ber of variables exceeds > 10000 and the number of outputs exceeds > 1000, as 
in typical genome-wide association studies. Furthermore, since the estimation 
error of the regression coefficients depends on the number of variables in the 
problem, variable selection can improve convergence rates of estimation proce- 
dures. These concerns motivate us to propose and study the S-OMP as a fast 
way to remove irrelevant variables from an ultra-high dimensional space. 

Formally, the GWA mapping problem, which we will use as an illustrative 
example both in here for model formulation and later for empirical experimental 
validation, can be cast as a variable selection problem in a multiple output 
regression model: 

Y = XB + W (1) 

where Y = [yi, . . . , y^] G M"^'^ is a matrix of outputs, whose column y^ is an 
n-vector for the t-th output (i.e., gene), X e R"^^ is a random design matrix, 
of which each row denotes a p-dimensional input, B = [/3i, . . . , (3t] G R^^"^ 
is the matrix of regression coefficients and W = [ei, . . . , ex] € R"^^ is a matrix 
of IID random noise, independent of X. Throughout the paper we are going to 
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assume that the columns of B are jointly sparse, as we precisely specify below. 
Note that if different columns of B do not share any underlying structure, the 
model in ([T]) can be estimated by fitting each of the tasks separately. 

We are interested in estimating the regression coefficients, under the as- 
sumption that they share a common structure, e.g., there exist a subset of 
variables with non-zero coefficients for more than one regression output. We 
informally refer to such outputs as related. Such a variable selection problem 
can be formalized in tw o ways: (1) the union support recovery of B, as defined in 
lObozinski et al. where a subset of variables is selected that affect at least 

one output; (2) the exact support recovery of B, where the exact positions of non- 
zero elements in B are estimated. In this paper, we concern ourselves with exact 
supp ort recovery, which is of particular importance in proble ms like GWA map - 
ping ( Kim and Xing. 2009() or biological network estimation ( Peng et al.. 2008 ). 



Under such a multi-task setting, two interesting questions naturally follow: i) 
how can information be shared between related outputs in order to improve 
the predictive accuracy and the rate of convergence of the estimated regression 
coefficients over the independent estimation on each output separately; ii) how 
to select relevant variables more accurately based on information from related 
outputs. To address these two questi ons, one line of research fe.g.. lZhang. 2006 



Liu et al.. 20091 : Lounici et al.. 20091) has looked into the following estimation 



procedure leveraging a multi-task regularization: 

T p 

B= argmin V ||yt - X/3t + A V pen(/3i . . . , /3tj), (2) 

/3teKP,tG[T] j^;^ ^-^1 



with pen(ai,...,aT) = maxjgfT] |at| or pen(ai, . . . , ot) = ^JJ2telT]"'t a 

vector a G M.'^ . Under an appropriate choice of the penalty parameter A, the es- 
timator B has many rows equal to zero, which correspond to irrelevant variables. 
However, solving ^ can be computationally prohibitive. 

In this paper, we consider an ultra-high dimensional setting for the afore- 
mentioned multi-task regression problem, where the number of variables p is 
much higher than the sample size n, e.g. p — 0{exp{n^p)) for a positive con- 
stant 6p, but the regression coefficients (3t are sparse, i.e., for each task t, there 
exist a very small number of variables that are relevant to the output. Under 
the sparsity assumption, it is highly important to efficiently select the relevant 
variables in order to improve the accuracy of the estimation and prediction, and 
to facilitate the understandin g of the underlyin g phenomenon for domain ex- 



perts. In the seminal paper of lFan and Lv ('20081 ). the concept of sure screening 



was introduced, which leads to a sequential variable selection procedure that 
keeps all the relevant variables with high probability in ultra-high dimensional 
uni-output regression. In this paper, we propose the S-OMP procedure, which 
enjoys sure screening property in ultra-high dimensional multiple output regres- 
sion as defined in ([l}. To perform exact support recovery, we further propose 
a two-step procedure that first use S-OMP to screen the variables, i.e., select a 
subset of variables that contain all the true variables; and then use the adap- 
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tive Lasso (ALasso) (|Zou. 20061 ) to further select a subset of screened variables 
for each task. We show, both theoretically and empirically, that our procedure 
ensure sparsistant recovery of relevant variables. To the best of our knowledge, 
this is the first attempt to analyze the sure screening property in the ultra-high 
dimensional space using the shared information from the multiple regression 
outputs. 



1.1 Related Work 

The model given in ([1]) has been used in many different domains ranging from 
multivariate regression (lObozinski et al.. 20091 : Negahban and \yainwright, 20091) 
and sparse approxim ation ( Tropp et al.. 20061) to neural science ( Liu et al.. 20091) . 
multi-task learning ( Lounici et al.. 20091 Argvriou et al.. 20081) and biological 
network estimation ( Peng et al.. 20081) . A number of authors has provided the- 
oretical understan ding of the estimation in the model using the convex program 



([2]) to estimate B. Lounici et al. (20"09l ) showed the benefits of the joint estima- 



tion, when there is a small set of variables c om mon to all outputs and the number 

of outputs is large. Obozinski et al. f2009l ) and Negahban and Wainwright f200g ) 

analyzed the consistent recovery of the union support. Negahban and Wainwright (2009t ) 
provided the analysis of the exact support recovery for a special case with two 
outputs. 

The Orthogona l Matching Pursuit (QMP ) has been analyzed befo re in the 

literature (see, e.g.. lZhang. 2009HLozano et al. 2009; Wang. 2009,; Barron et al. 20081) . 
In particular, our work should be contrasted to I Wang (20091 ). which showed that 
the OMP has the sure screening property in a linear regression with a single 
output, and to th e exact variable sele ction property of the OMP analyzed in 
Zhang r2no9[ ) and iLozano et al. The exact variable selection requires 

much stronger assumptions on the design, such as the irrepresentable condition, 
that are hard to satisfy in the ultra-high dimensional setting. On the other 
hand, the sure screening property can be shown to hold under much weaker 
assumptions. 

In this paper, we make the following novel contributions: i) we prove that 
the S-OMP can be used for the ultra-high dimensional variable screening in mul- 
tiple output regression problems and demonstrate its performance on extensive 
numerical studies; ii) we show that a two step procedure can be used to select 
exactly the releva nt variables for each tas k; and iii) we prove that a modification 
of the BIG score ( Chen and Chen. 20081) can be used to select the number of 
steps in the S-OMP. 

The rest of the article is organized as follows. In Section 2, we introduce the 
simultaneous greedy forward regression and propose our approach to the exact 
support estimation. Theoretical guarantees of the methods are given in Section 
3. Section 4 is devoted to extensive numerical simulations. An application to 
the real world problem in association mapping is demonstrated in Section 5. 
We conclude with discussion in Section 6. Proofs are deferred to Appendix. 
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2 Methodology 



2.1 The model and notation 

We will consider a slightly more general model 

yi = Xi/3i + ei 
y2 = X2/32 + £2 

Yt ~ ^rf^T + ^T, 

than the one given in ([IJ. The model in (jlj is a special case of the model 
in with all the design matrices {Xt}fg[y] being equal and [T] denoting 
the set {1,...,T}. Assume that for all t e [T], e K"^p. For the design 
Xt, we denote Xj^j the j-th column (i.e., dimension), Xt_i the i-th row (i.e., 
instance) and Xt^ij the element at (i, j). Denote Si = Cov(x(^i). Without loss 
of generality, we assume that Yaj:{yt^i) = 1, ^{xt^ij) = and YaT{xt,ij) — 1. 
The noise e* is zero mean and Cov(et) = a'^lnxn- We assume that the number 
of variables p ^ n and that the vector of regression coefficients f3t are jointly 
sparse, that is, there exist a small number of variables that are relevant for 
the most of the T regression problems. Put another way, the matrix B = 
[/3i, . . . ,/3t] has only a small number of non-zero rows. Let A^*,t denote the set 
of non-zero coefficients of (3t and A^* = yJ^=iM.*.t denote the set of all relevant 
variables, i.e., variables with non-zero coefficient in at least one of the tasks. For 
an arbitrary set C {1, . . . 'K.t^M denotes the design with columns indexed 
by A^, denotes the rows of B indexed by M and Bj = (/3i.ji • • ■ 1 f^T.j)' ■ 
The cardinality of the set A4 is denoted as \A4\. Let s :— |A^*| denote the 
total number of relevant variables, so under the sparsity assumption we have 
s < n. For a square matrix A, Aiuin(A) and Amax(A) are used to denote the 
minimum and the maximum eigenvalue, respectively. For a different matrix 
A = [uij] e RP^^, we define ||A||2,i := Y.^e[p] \/j2je[T] ^tj- Lastly, we use [p] 
to denote the set {1, . . . ,p}. 

2.2 Simultaneous Orthogonal Matching Pursuit 

We propose a Simultaneous Orthogonal Matching Pursuit procedure for ultra 
high-dimensional variable selection in the multi-task regression problem, which 
is outlined in Algorithm [T] Before describing the algorithm, we introduce some 
additional notation. For an arbitrary subset A4 C [p] of variables, let tlt,M be 
the orthogonal projection matrix onto Span(Xt_7vi), i.e., 

Ht^Ai = X(_7vi(Xj _/^Xt^^) (4) 

and define the residual sum of squares (RSS) as 

T 

RSS(A^) =^y;(I„x„-HA,)yt. (5) 

t=i 



5 



Algorithm 1 Group Forward Regression 



Input: Dataset {'Kt,'yt}T=i 

Output: Sequence of selected models {M^''^}]^Zq 

1: Set = 

2: for fc = 1 to n — 1 do 

3: for j = 1 to p do 

5: Htj = X^_^(fc)(X[^^(,)X^^^(,))-iX' 
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7: 
8 

9 
10 



RSS(>lf' ) = ELi yUlnxn - Hi,,)yi 
end for 

fk = argmin^.g^j_ pj^^(fe_i, RSS(7Wf ) 
end for 



The algorithm starts with an empty set A^*^'^' 0. We recursively define 
the set M^'^^ based on the set A^C^"^). The set A^C^) is obtained by adding 
a variable indexed by fk € [p], which minimizes U j) over the set 

[p]\M^''^^\ to the set Al^'^^^'. Repeating the algorithm for n — 1 steps, a 
sequence of nested sets {M^'^^'^^o obtained, with A^C"') — {/i, . . . , fk}. 

To practically select one of t he sets of variables from {A^'-'^-'I^Zq , we mini- 



mize the modified BIG criterion (Ghen and Ghen. 2008f). which is defined as 



BIG(A.) = log (^^) + |A^iaosH + 21og(,)) 
\ nT J n 

with \A4\ denoting the number of elements of the set Ai. Let 

s — argmin BIG(A^('')), 

fcG{0,...,n-l} 

SO that the selected model is A^*^"). 

Remark: The S-OMP algorithm is outlined only conceptually in this sec- 
tion. The steps 5 and 6 of the algorithm can b e implemented eflncie ntly using the 
progressive Gholesky decomposition see, e.g.. iGotter et al. C1999I) . A computa- 



tionally costly step 5 involves inversion of the matrix X^^Xtjn, however, it 
can be seen from the algorithm that the matrix Xj ^Xt^^vf is updated in each it- 
eration by simply appending a row and a column to it. Therefore, its Gholesky 
factorization can be efficiently computed based on calculation involving only 
the last row. A detailed implementation of the orthogonal matching pursuit 
algorithm based on the progressive Gholesky decomposition can be found in 



Rubinstein et al. f2008f ). 



2.3 Exact variable selection 

After removing many of the irrelevant variables have been removed using Al- 
gorithm [1] we are left with the variables in the set A^^'*^ whose size is smaller 
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than the sample size n. These variables are candidates for the relevant variables 
for each of the regressions. Now, we can address the problem of estimating 
the regression coefficients and recovering the exact support of B using a lower 
dimensional selection procedure. In this paper, we use the adaptive Lasso as a 
lowe r dimension al selection procedure, which was shown to have oracle proper- 
ties ( Zou. 20061) . The ALasso solves the penalized least square problem 



0t = argmin ||yt - Xt^M(s)Pt\\l + ^ ^ ^i'jIA.jl, (7) 



where iwj)jfzj^(s) is a vector of known weight and A is a tuning parameter. 
Usually, the weights are defined as Wj — l/\j3tj\ where (3 t is a -y/n-consistent 



estimator of Pt- In a low dimensional setting, we know from lHuang et al. f2008f ) 
that the adaptive Lasso can recover the exactly the relevant variables. Therefore, 
we can use the ALasso on each output separately to recover the exact support 
of B. However, in order to ensure that the exact support of B is recovered with 
high probability, we need to have that the total number of tasks is o(n). The 
exact support recovery of B is established using the union bound over differ- 
ent tasks, therefore we need the number of tasks to remain relatively small in 
comparison to the sample size n. However, simulation results presented in § ref- 
sec:simulation show that the ALasso procedure succeeds in the exact support 
recovery even when the number of tasks are much larger than the sample size, 
which indicates that our theoretical considerations could be improved. Figure [1] 
illustrates the two step procedure. 



Full Model 


S-OMP 




Only relevant variables 


ALasso 


Exact support 


large number of variables 


Screening 




small number of variables 


^ 



Figure 1: Framework for exact suppor recovery 

Remark: We point out that solving the multi-task problem defined in ([2]) 
can be efficiently done on the reduced set of variables, but it is not obvious 
how to obtain the estimate of the exact support using ([2]). In Section H?T1 our 
numerical studies show that the ALasso applied to the reduced set of variables 
can be used to estimate the exact support of B. 



3 Theory 

In this section we state conditions under which Algorithm [T] is screening consis- 
tent, i.e., 

P[3fc e {0, 1, . . . , n - 1} ; A^, C TW^^)] -> 1, as n -> oo. (8) 

Furthermore, we also show that the model selected using the modified BIC 
criterion contains all the relevant variables, i.e., 

P[M* C M'^'^] -^1, as n cxD. (9) 
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Note that we can choose triviahy A^(") since it holds that Ai* C How- 
ever, we wih be able to prove that s chosen by the modified BIC criterion is 
much smaller than the sample size n. 



3.1 Assumptions 

Before we state the theorem characterizing the performance of the S-OMP, we 
give some technical conditions that are needed for our analysis. 

Al: The random noise vectors ei, . . . , are independent Gaussian with zero 
mean and covariance matrix cr^I„xn- 

A2: Each row of the design matrix Xt is IID Gaussian with zero mean and 
covariance matrix Hf Furthermore, there exist two positive constants 

< 0min < '/'max < OO SUCh that 

4'min < min Ami„(St) < max A„iax(St) < (/imax- (10) 

te[T] te[T] 



A3: The true regression coefficients are bounded, i.e., there exists a positive 
constant C/g such that ||B||2,i < Cp. Furthermore, the norm of any non- 
zero row of the matrix B is bounded away from zero, that is, there exist 
positive constants Cf3 and S^i^ such that 

min y > Cfln~''»'". 

te[T] 



A4: There exist positive constants Cs,Cp,ds and Sp such that |A^*| < C< 
and log(p) < CpTL^p. 



The normality condition Al is assumed here only to facilit ate presentation of 



theoretica l results, as is commonly assumed in literature, (e.g.. lZhang and Huang. 2008 



Fan and Lv . 2008) . The n ormality assumption can be avoided at the cost of 



more technical proofs, e.g.. lLounici et al. (20091) ■ where the main technical diffi- 



culty is showing that the concentration properties still hold. Under the condition 
A2 we will be able to show that the empirical covariance matrix satisfies the 
sparse eigenvalue condition (see Lemma[3]) with probability tending to one. The 
assumption that the rows of the design are Gaussian can be easily relaxed to 
the case when the rows are sub-Gaussian, without any technical difficulties in 
proofs, since we would still obtain exponential bounds on the tail probabilities. 
The condition A3 states that the regression coefficients are bounded, which is a 
technical condition likely to be satisfied in practice. Furthermore, it is assumed 
that the row norms of "^m, do not decay to zero too fast or, otherwise, they 
would not be distinguishable from noise. The condition is not too restrictive, 
e.g., if every non-zero coefficient is bounded away from zero by a constant, the 
condition A3 is trivially satisfied with (5niin — 0. However, we allow the coeffi- 
cients of the relevant variables to get smaller as the sample size increases and 
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still guarantee that the relevant variable will be identified. The condition A4 
sets the upper bound on the number of relevant variables and the total number 
of variables. While the total number of variables can diverge to infinity much 
faster than the sample size, the number of relevant variables needs to be smaller 
than the sample size. Conditions A3 and A4 implicitly relate different out- 
puts and control the number of non-zero coefficients shared between different 
outputs. Intuitively, if the upper bound in A4 on the size of A^* is large, this 
immediately implies that the constant Cp in A3 should be large as well, since 
otherwise there would exist a row of B whose £2 norm would be too small to be 
detected by Algorithm [1] 



3.2 Screening consistency 

Our first results states that after a small number of iterations, compared to the 
dimensionality p, the S-OMP procedure will include all the relevant variables. 

Theorem 1. Assume the model in ([3]) and that the conditions A1-A4 are 
satisfied. Furthermore, assume that 



max{log(p),log(r)} 



00, as n ^ 00. (11) 



Then there exists a number ni'^^^^ — m'^^^{n), so that in m*^g^^ steps of S-OMP 
iteration, all the relevant variables are included in the model, i.e., as n —> 00 

/ l-65s-6<5,„i„ \ 

V max{log(p),log(T)}y 
for some positive constants Ci and C2 • The exact value of m'^^^ is given as 

Remarks: Under the assumptions of Theorem [l] m*^^^^ < n ~ 1, so that the 
procedure effectively reduces the dimensionality below the sample size. From 
the proof of the theorem, it is clear how multiple outputs help to identify the 
relevant variables. The crucial quantity in identifying all relevant variables is 
the minimum non-zero row norm of B, which allows us to identify weak vari- 
ables if they are relevant for a large number of outputs even though individual 
coefficients may be small. It should be noted that the main improvement over 
the ordinary forward regression is in the seize of the signal that can be detected, 
as defined in A3 and A4. 

Theorem [1] guarantees that one of the sets {A^*-'^-'} will contain all relevant 
variables, with high probability. However, it is of practical importance to select 
of one set in the collection that contains all relevant variables and does not 
have too many irrelevant ones. Our following theorem shows that the modified 
BIG criterion can be used for this purpose, that is, the set A^'*^) is screening 
consistent. 
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Theorem 2. Assume that the conditions of Theorem]^ are satisfied. Let 

s= argmin BIC(A^(''^) (14) 

ke{0,...,n-l} 

he the index of the model selected by optimizing the modified BIC criterion. 
Then, as n ^ co 

P[M, C m'-'^ 1. (15) 

Combining results from Theorem [T] and Theorem [5J we have shown that the 
S-OMP procedure is screening consistent and can be apphed to problems where 
the dimensionality of the problem p is exponential in the number of observed 
samples. In the next section, we also show that the S-OMP has great empirical 
performance. 



4 Numerical studies 

In this section we perform simulation studies on an extensive number synthetic 
data sets. Furthermore, we demonstrate the application of the procedure on the 
genome- wide association mapping problem. 



4.1 Simulation studies 

We conduct an extensive number of numerical studies to evaluate the finite 
sample performance of the S-OMP. We consider three procedures that perform 
estimation on individuals outp uts: Sure Independence Screening (SIS), Iterative 
SIS (ISIS) (|Fan and Lv. 20081) . and the OMP, for comparison purposes. The 



evaluation is done on the model in ([T]) . SIS and ISIS are used to select a subset of 
variables and then the ALasso is used to further refine the selection. We denote 
this combination as SIS- ALasso and ISIS- ALasso. The size of the model selected 
by SIS is fixed as n — 1, while the ISIS selects [n/ log(n)J variables in each of the 
[log(n) — Ij iterations. From the screened variables, the final model is selected 
using the ALasso, together with the BIC criterion ([5]) to determine the penalty 
parameter A. The number of variables selected by the OMP is determined using 
the BIC criterion, however, we do not further refi ne the sele cted variables using 
the ALasso, since from the numerical studies in IWang f200a) it was observed 
that the further refinement does not result in improvement. The S-OMP is used 
to reduce the dimensionality below the sample size jointly using the regression 
outputs. Next, the ALasso is used on each of the outputs to further perform 
the estimation. This combination is denoted SOMP- ALasso. 

Let B = [/3i, . . . ,/3t] € M^"^-^ be an estimate obtained by one of the esti- 
mation procedures. We evaluate the performance averaged over 200 simulation 
runs. Let ]E„ denote the empirical average over the simulation runs. We measure 
the size of the union support S = 5'(B) := {j G [p] : ||Bj||| > 0}. Next, we esti- 
mate the probability that the screening property is satisfied E„[1I{A^* C 5(6)}], 
which we call coverage probability. For the union support, we define frac- 
tion of correct zeros {p — s)~^]E„[|S'(B)'-^ n A^f |], fraction of incorrect zeros 
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s-iE„[|S'(B)<^ n and fraction of correctly fitted E„[]I{7W* = S{B)}] to 

measure the performance of different procedures. Similar quantities are defined 
for the exact support recovery. In addition, we measure the estimation error 
E„[||B — B|||] and the prediction performance on the test set. On the test data 
{^hy*}ie[n], we compute 

„2 1 T,ie[n]T,te[T]iyli - i'^t,iy f^tf 

" r r (v* -v*V ' ^ ^ 

where = -nT^ XiieH ^Z*-*' 

The following simulation studies are used to comparatively assess the nu- 
merical performance of the procedures. Due to space constraints, tables with 
detailed numerical results are given in the appendix. In this section, we outline 
main findings. 

Simulation 1: [Model w ith uncorrelated va, riables1 The following toy model is 
based on the simulation I in lFan and Lv (20081) with (n,p,s,r) = (400,20000,18,500). 
Each Xi is drawn independently from a standard multivariate normal distribu- 
tion, so that the variables are mutually independent. For j G [s] and t e [T], the 
non-zero coefficients of B are given as Pt^j = (-l)"(4n-i/2iogn + l^l), where 
u Bernoulli(0.4) and z ^ A/'(0, 1). The number of non-zero elements in B^- 
is given as a parameter Tnon-zcro G {500,300, 100}. The positions of non-zero 
elements are chosen uniformly at random from [T] . The noise is Gaussian with 
the standard deviation a set to control the signal-to- noise ratio (SNR). SNR is 
defined as Var(x/3)/ Var(e) and we vary SNR e {15, 10, 5, 1}. 

Simulation 2: [Changing the number of non-zero elements in Bj] The follow- 
ing scenario is used to evaluate the performance of the methods as the number 
of non-zero elements in a row of B varies. We set {n,p, s) = (100, 500, 10) and 
vary the number of outputs T e {500,750,1000}. For each number of out- 
puts T, we vary Tnon-zoro € {0.8T, 0.5T, 0.2T}. The samples x^ and regression 
coefficients B are given as in Simulation 1, i.e., x^ is drawn from a multivari- 
ate standard normal distribution and the non-zero coefficients B are given as 
/3j_^ = (-l)«(47i-i/2 logn+ |z|), where u - Bernoulfi(0.4) and z ~ 7V(0, 1). The 
noise is Gaussian, with the standard deviation defined through the SNR, which 
varies in {10, 5, 1}. 

Simulation 3: [Model with th e decaying co rrelation between variables] The 
following model is borrowed from I Wang (20091 ). We assume a correlation struc- 
ture between variables given as Var(Xj,, J = where pG {0.2,0.5,0.7}. 
This correlation structure appears naturally among ordered variables. We set 
{n,p,s,T) — (100,5000,3,150) and Tnon-zcro — 80. The relevant variables are 
at positions (1,4,7) and non-zero coefficients are given as 3, 1.5 and 2 respec- 
tively. The SNR varies in {10,5,1}. A heat map of the correlation matrix 
between different covariates is given in Figured) 

Simulation J^: [Model with the block-compound correlation structure] The 
following model assumes a block compound correlation structure. For a param- 
eter p, the correlation between two variables X^-^ and X^-^ is given as p, (? or 
p3 when |ji - jal < 10, |ji - h\ e (10,20] or |ji - jsl £ (20,30] and it is set 
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■ -u. 

I 



(a) p = 0.2 



(b) p = 0.7 



Figure 2: Visualization of the correlation matrix in Simulation 3. Only an upper 
left corner is presented corresponding to 20 of the 5000 variables. 



to otherwise. We set {n,p,s,T) = (150,4000,8, 150), T„o„-zero = 80 and the 
parameter p £ {0.2,0.5}. The relevant variables are at located at positions 1, 
11, 21, 31, 41, 51, 61, 71 and 81, so that each block of highly correlated variables 
has exactly one relevant variable. The values of relevant coefficients is given as 
in Simulation 1. The noise is Gaussian and the SNR varies in {10, 5, 1}. A heat 
map of the correlation matrix between different covariates is given in Figure [3l 





50 60 70 aO 90 100 



(a) p = 0.2 



(b) p = 0.5 



Figure 3: Visualization of the correlation matrix in Simulation 4. Only an upper 
left corner is presented corresponding to 100 of the 4000 variables. 



Simulation 5: [Model with a 'masked' r elevant varia, ble] This model repre- 
sents a difficult setting. It is modified from Wang ('2009() . We set {n,p,s,T) = 
(200,10000,5,500). The number of non-zero elements in each row varies is 
Tkon-zcro G {400,250,100}. For j e [s] and t G [T], the non-zero elements 
equal (3tj ~ 2j. Each row of X is generated as follows. Draw independently Zj 
and z[ from a p-dimensional standard multivariate normal distribution. Now, 
Xij = {z^j + z[^)l^(2) for i e [s] and Xij = (zy -I- E/eW %-')/2 for j € [p]\[s]. 
Now, Corr(a;i.i, yt^i) is much smaller then Corr(a;ij-, yt^i) for j G [p]\[s], so that it 



12 



becomes difficult to select variable 1. The variable 1 is 'masked' with the noisy 
variables. This setting is difficult for screening procedures as they take into 
consideration only marginal information. The noise is Gaussian with standard 
deviation a G {1.5, 2.5, 4.5}. 

In the next section we summarize results of our experimental findings. Our 
simulation setting transitions from a simple scenario considered in Simulation 
1 towards a chall enging one in Simulation 5. Simulation 1 is adopted from 



Fan and Lv (^20081 ) as a toy model on which all algorithms should work well. 



Simulation 2 examines the influence of the number of non-zero elements in 
a relevant row of the matrix B. We expect that Algorithm [T] will outper- 
form procedures that perform estimation on individual outputs when Tnon-zoro 
is large, while when Tnon-zcro is small the single-task screening procedures 
sh ould have an adva ntage. Our intuition is also supported by recent results 
of Kolar et al. Simulations 3 and 4 represent more challenging situa- 

tions with structured correlation that naturally appears in many data sets, for 
example, a correlation between gene measurements that are closely located on a 
chromosome. Finally Simulation 5 is constructed in such a way that procedures 
which use only marginal information are going to include irrelevant variables 
before relevant ones. 



4.2 Results of simulations 

Tables giving detailed results of the above described simulations are given in the 
Appendix. In this section, we outline main findings and reproduce some parts 
of the tables that we think are insightful. 

Table [T] shows parts of the results for simulation 1. We can see that all 
methods perform well in the setting when the input variables are mutually 
uncorrelated and the SNR is high. Note that even though the variables are 
uncorrelated, the sample correlation between variables can be quite high due to 
large p and small n, which can result in selection of spurious variables. As we can 
see from the table, comparing to SIS, ISIS and OMP, the S-OMP is able to select 
the correct union support, while the procedures that select variables based on 
different outputs separately also include additional spurious variables into the 
selection. Furthermore, we can see that the S-OMP-ALasso procedure does 
much better on the problem of exact support recovery compared to the other 
procedures. The first simulations suggests that somewhat higher computational 
cost of the S-OMP procedure can be justified by the improved performance on 
the problem of union and exact support recovery as well as on the error in the 
estimated coefficients. 

Table [2] shows parts of the results for simulation 2. In this simulation we 
measured the performance of estimation procedures as the amount of shared 
input variables between different outputs varies. The parameter Tnon-zcro con- 
trols the amount of information that is shared between different tasks as defined 
in the previous subsection. In particular, the parameter controls the number of 
non-zero elements in a row of the matrix B corresponding to a relevant variable. 
When the number of non-zero elements is high, a variable is relevant to many 
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Table 1: Results for simulation 1 with parameters {n,p, s, T) — (500, 20000, 18, 500), Tnon-zero = 500 







Prob. (%) of 


Fraction (%) of 


Fraction (%) of 


Fraction (%) of 




Est. error 


Test error 




Method name 




Correct zeros 


Incorrect zeros 


M^=S 


\s\ 


l|B-B||| 


R? 










SNR = 15 












SIS-ALASSO 


100.0 


100.0 


0.0 


10.0 


20.2 






^ g 


ISIS-ALASSO 


100.0 


100.0 


0.0 


18.0 


19.6 








OMP 


100.0 


100.0 


0.0 


0.0 


23.9 








S-OMP 


100.0 


100.0 


0.0 


100.0 


18.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


18.0 






+^ 


SIS-ALASSO 


0.0 


100.0 


0.7 


0.0 


8940.5 


0.97 


0.93 




ISIS-ALASSO 


100.0 


100.0 


0.0 


18.0 


9001.6 


0.33 


0.93 




OMP 


100.0 


100.0 


0.0 


0.0 


9005.9 


0.20 


0.93 




S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


9000.0 


0.20 


0.93 



Table 2: Results for simulation 2 with parameters (n,p, s, T) = (200, 5000, 10, 1000), Tnon-zero = 200 







Prob. (%) of 


Fraction (%) of 


Fraction (%) of 


Fraction (%) of 




Est. error 


Test error 




Method name 


M, CS 


Correct zeros 


Incorrect zeros 


M,=S 


\s\ 


l|B-B||i 












SNR = 5 












SIS-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 






u 

^ 2 


ISIS-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 






o 


OMP 


100.0 


97.4 


0.0 


0.0 


139.6 






^ & 

^ zn 


S-OMP 


100.0 


100.0 


0.0 


100.0 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 






+^ 


SIS-ALASSO 


100.0 


100.0 


0.0 


100.0 


2000.0 


0.04 


0.72 


4J 

o a 


ISIS-ALASSO 


100.0 


100.0 


0.0 


100.0 


2000.0 


0.04 


0.72 




OMP 


100.0 


100.0 


0.0 


0.0 


2131.6 


0.05 


0.71 


Hen 


S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


2000.0 


0.03 


0.72 



Table 3: Results for simulation 3 with parameters (n,p, s, T) = (100, 5000, 3, 150), Tnon-zero = 80, p = 0.5 





Method name 


Prob. (%) of 

M*CS 


Fraction (%) of 
Correct zeros 


Fraction (%) of 
Incorrect zeros 


Fraction (%) of 

M*=S 


\s\ 


Est. error 
l|B-B||i 


Test error 










SNR = 5 










Union 
Support 


SIS-ALASSO 
ISIS-ALASSO 
OMP 
S-OMP 
S-OMP-ALASSO 


100.0 
100.0 
100.0 
100.0 
100.0 


100.0 
100.0 
99.8 
100.0 
100.0 


0.0 
0.0 
0.0 
0.0 
0.0 


97.0 
96.0 
0.0 
100.0 
100.0 


3.0 
3.0 
19.6 
3.0 
3.0 






Exact 
Support 


SIS-ALASSO 
ISIS-ALASSO 
OMP 
S-OMP-ALASSO 


60.0 
84.0 
100.0 
100.0 


100.0 
100.0 
100.0 
100.0 


0.2 
0.1 
0.0 
0.0 


57.0 
80.0 
0.0 
100.0 


239.5 
239.8 
256.6 
240.0 


0.10 
0.08 
0.06 
0.03 


0.61 
0.61 
0.61 
0.62 
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Table 4: Results of simulation 4 with parameters {n,p, s, T) = (150, 4000, 8, 150), Tnon-zero = 80, p = 0.5 





Method name 


Prob. (%) of 
M,QS 


Fraction (%) of 
Correct zeros 


Fraction (%) of 
Incorrect zeros 


Fraction (%) of 
M,=S 


\s\ 


Est. error 
l|B-B||i 


Test error 










SNR = 10 










Union 
Support 


SIS-ALASSO 
ISIS-ALASSO 
OMP 
S-OMP 
S-OMP-ALASSO 


100.0 
100.0 
100.0 
100.0 
100.0 


100.0 
100.0 
99.9 
100.0 
100.0 


0.0 
0.0 
0.0 
0.0 
0.0 


100.0 
97.0 
2.0 

100.0 
100.0 


8.0 
8.0 
11.7 
8.0 
8.0 






Exact 
Support 


SIS-ALASSO 
ISIS-ALASSO 
OMP 
S-OMP-ALASSO 


35.0 
100.0 
100.0 
100.0 


100.0 
100.0 
100.0 
100.0 


1.4 
0.0 
0.0 
0.0 


35.0 
97.0 
2.0 
100.0 


631.3 
640.0 
643.7 
640.0 


0.55 
0.14 
0.10 
0.09 


0.88 
0.89 
0.89 
0.89 



Table 5: Results of simulation 5 with parameters (n,p, s, T) = (200, 10000, 5, 500), Tnon-zero = 400 





Method name 


Prob. (%) of 
M,QS 


Fraction (%) of 
Correct zeros 


Fraction (%) of 
Incorrect zeros 


Fraction (%) of 
M*=S 


1^1 


Est. error 
l|B-B||i 


Test error 


Union 
Support 


SIS-ALASSO 
ISIS-ALASSO 
OMP 
S-OMP 
S-OMP-ALASSO 


53.0 
100.0 
100.0 
100.0 
100.0 


99.6 
99.8 
99.9 
100.0 
100.0 


a = 1.5 

9.4 
0.0 
0.0 
0.0 
0.0 


0.0 
0.0 
12.0 
44.0 
100.0 


41.1 
28.1 
10.0 
5.6 
5.0 






Exact 
Support 


SIS-ALASSO 
ISIS-ALASSO 
OMP 
S-OMP-ALASSO 


0.0 
0.0 
100.0 
100.0 


100.0 
100.0 
100.0 
100.0 


68.9 
16.2 
0.0 
0.0 


0.0 
0.0 

12.0 
100.0 


936.0 
1791.9 
2090.3 
2000.0 


84.66 
5.80 
0.06 
0.05 


0.66 
0.96 
0.99 
0.99 
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tasks and we say that outputs overlap. In this setting the S-OMP procedure 
is expected to outperform the other methods, however, when Tnon-zero is fow, 
the noise coming from the tasks for which the variable is irrelevant can actually 
harm the performance. The table shows results when the overlap of shared vari- 
ables is small, that is, a relevant variable is only relevant for 10% of outputs. 
As one could expect, the S-OMP procedure does as well as other procedures. 
This is not surprising, since the amount of shared information between differ- 
ent outputs is limited. Therefore, if one expects little variable sharing across 
different outputs, using the SIS or ISIS may results in similar accuracy, but an 
improved computational efhciency. It is worth pointing out that in our simu- 
lations, the different tasks are correlated since the same design X used for all 
tasks. However, we expect the same qualitative results even under the model 
given in equation ([3]) where different tasks can have different designs and 
the outputs are uncorrelated. 

Simulation 3 represents a situation that commonly occur in nature, where 
there is an ordering among input variables and the correlation between variables 
decays as the distance between variables increase. The model in simulation 4 
is a modification of the model in simulation 3 where the variables are grouped 
and there is some correlation between different groups. Table [3] gives results 
for simulation 3 for the parameter p = 0.5. In this setting, the S-OMP per- 
forms much better that the other procedures. The improvement becomes more 
pronounced with increase of the correlation parameter p. Similar behavior is 
observed in simulation 4 as well, see table ID Results of simulation 5, given in 
Table[5l further reinforce our intuition that the S-OMP procedure does well even 
on problems with high-correlation between the set of relevant input variables 
and the set of irrelevant ones. 

To further compare the performance of the S-OMP procedure to the SIS, we 
explore the minimum number of iterations needed for the algorithm to include 
all the relevant variables into the selected model. From our limited numerical 
experience, we note that the simulation parameters do not affect the number 
of iterations for the S-OMP procedure. This is unlike the SIS procedure, which 
occasionally requires a la rge number of step s before all the true variables are 
included, see Figure 3 in lFan and Lv f 20081 ). We note that while the S-OMP 
procedure does include, in many cases, all the relevant variables before the 
irrelevant ones, the BIG criterion is not able to correctly select the number of 
variables to include, when the SNR is small. As a result, we can see the drop 
in performance as the SNR decreases. 



4.3 Real data analysis 

We demonstrate an application of the S-OMP to a genome-wide association 
mapping problem. The data were collected by our collaborator Judie Howry- 
lak, M.D. at Harvard Medical School from 200 individuals that are suffering 
from asthma. For each individual, we have a collection of about ~ 350000 ge- 
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netic marker^, which are cahed single nucleotide polymorphisms (SNPs), and 
a collection of 1424 gene expression measurements. The goal of this study is 
to identify a small number of SNPs that can help explain variations in gene 
expressions. Typically, this type of analysis is done by regressing each gene in- 
dividually on the measured SNPs, however, since the data are very noisy, such 
an approach results selecting many variables. Our approach to this problem 
here is to regress a group of gen es onto the SNPs inst ead. There has been some 
previous work on this problem iKim and Xing f 20091) ■ that considered regress- 
ing groups of genes onto SNPs, however, those approaches use variants of the 
estimation procedure given in Eq. ([2]), which is not easily scalable to the data 
we analyze here. 

We use the spectral relaxation of the k-means clustering lZha et al. (20011 ) to 
group 1424 genes into 48 clusters according to their expression values, so that the 
minimum, maximum and median number of genes per cluster is 4, 90 and 19, 
respectively. The number of clusters was chosen somewhat arbitrarily, based 
on the domain knowledge of the medical experts. The main idea behind the 
clustering is that we want to identify genes that belong to the same regulatory 
pathway since they are more likely to be affected with the same SNPs. Instead of 
clustering, one may use prior knowledge to identify interesting groups of genes. 
Next, we want to use the S-OMP procedure to identify relevant SNPs for each 
of the gene clusters. Since, we do not have the ground truth for the data set, we 
use predictive power on the test set and the size of estimated models to access 
their quality. We randomly split the data into a training set of size 170 and 
a testing set of size 30 and report results over 500 runs. We compute the 
coefficient on the test set defined as 1 — SO^^T^^ J2te[T] l|yt,tcst — Xt^tcstAHi 
(because the data has been normalized). 

Due to space constraints, we give results on few clusters in Table [6] and 
note that, qualitatively, the results do not vary much between different clusters. 
While the fitted models have limited predictive performance, which results from 
highly noisy data, we observe that the S-OMP is able to identify on average one 
SNP per gene cluster that is related to a large number of genes. Other methods, 
while having a similar predictive performance, select a larger number of SNPs 
which can be seen from the size of the union support. On this particular data set, 
the S-OMP seems produce results that are more interpretable from a specialist's 
points of view. Further investigation needs to be done to verify the biological 
significance of the selected SNPs, however, the details of such an analysis are 
going to be reported elsewhere. 



5 Conclusions 

In this work, we analyze the Simultaneous Orthogonal Matching Pursuit as 
a method for variable selection in an ultra-high dimensional space. We prove 
that the S-OMP is screening consistent and provide a practical way to select the 

^ These markers were preprocessed, by imputing missing values and removing duplicate 
SNPs that were perfectly correlated with other SNPs. 
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Table 6: Results on the asthma data 





Method name 


Union support 




Cluster 9 
Size = 18 


SIS-ALASSO 
OMP 
S-OMP 


18.0 (1.0) 
17.5 (2.9) 
1.0 (0.0) 


0.178 
0.167 
0.214 


(0.006) 
(0.002) 
[0.005) 


Cluster 16 
Size = 31 


SIS-ALASSO 
OMP 
S-OMP 


31.0 (1.0) 
29.0 (1.8) 
1.0 (0.0) 


0.160 
0.165 
0.209 


(0.007) 
(0.002) 
[0.005) 


Cluster 17 
Size = 19 


SIS-ALASSO 
OMP 
S-OMP 


18.5 (0.9) 
19.5 (0.8) 
1.0 (0.0) 


0.173 
0.146 
0.184 


(0.006) 
(0.003) 
[0.004) 


Cluster 19 
Size = 17 


SIS-ALASSO 
OMP 
S-OMP 


17.0 (1.2) 
11.0 (4.1) 
1.0 (0.0) 


0.270 
0.213 
0.280 


(0.017) 
(0.008) 
[0.017) 


Cluster 22 
Size = 34 


SIS-ALASSO 
OMP 
S-OMP 


34.0 (0.9) 
30.0 (7.3) 
1.0 (0.0) 


0.153 
0.142 
0.145 


(0.005) 
(0.000) 
[0.002) 


Cluster 23 
Size = 35 


SIS-ALASSO 
OMP 
S-OMP 


35.0 (0.9) 
33.0 (9.9) 
1.0 (0.0) 


0.238 
0.208 
0.229 


(0.018) 
(0.009) 
[0.014) 


Cluster 24 
Size = 28 


SIS-ALASSO 
OMP 
S-OMP 


28.0 (1.0) 
28.0 (2.6) 
1.0 (0.0) 


0.123 
0.114 
0.129 


(0.003) 
(0.001) 
[0.003) 


Cluster 32 
Size = 15 


SIS-ALASSO 
OMP 
S-OMP 


15.0 (0.9) 
10.0 (2.6) 
1.0 (0.0) 


0.188 
0.211 
0.215 


(0.010) 
(0.006) 
[0.008) 


Cluster 36 
Size = 33 


SIS-ALASSO 
OMP 
S-OMP 


34.0 (1.4) 
29.0 (5.3) 
1.0 (0.0) 


0.147 
0.157 
0.168 


(0.005) 
(0.002) 
(0.004) 


Cluster 37 
Size = 19 


SIS-ALASSO 
OMP 
S-OMP 


19.0 (0.9) 
22.0 (2.5) 
1.0 (0.0) 


0.207 
0.175 
0.235 


(0.015) 
(0.006) 
(0.014) 


Cluster 39 
Size = 24 


SIS-ALASSO 
OMP 
S-OMP 


24.0 (0.9) 
27.0 (1.9) 
1.0 (0.0) 


0.131 
0.141 
0.160 


(0.006) 
(0.003) 
(0.005) 


Cluster 44 
Size = 35 


SIS-ALASSO 
OMP 
S-OMP 


35.0 (0.9) 
26.5 (6.6) 
1.0 (0.0) 


0.177 
0.183 
0.170 


(0.010) 
(0.005) 
(0.011) 


Cluster 49 
Size = 23 


SIS-ALASSO 
OMP 
S-OMP 


23.0 (1.0) 
23.0 (1.2) 
1.0 (0.0) 


0.124 
0.140 
0.159 


(0.004) 
(0.000) 
(0.004) 
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number of steps in the procedure using the modified Bayesian information crite- 
rion. Our hmited numerical experience shows that the method performs well in 
practice and that the joint estimation from multiple outputs often outperforms 
methods that use one regression output at the time. Furthermore, we can see 
the S-OMP procedure as way to improve the variable selection properties of the 
SIS without having to solve a costly complex optimization procedure in Eq. 
therefore, balancing the computational costs and the estimation accuracy. 



6 Appendix 

6.1 Proof of Theorem [T] 

Under the assumptions of the theorem, the number of relevant variables s is 
relatively small compared to the sample size n. The proof strategy can be 
outlined as follows: i) we are going to show that, with high probability, at least 
one relevant variable is going to be identified within the following rn*^^^ steps, 
conditioning on the already selected variables Ai and this holds uniformly for 
all k; ii) we can conclude that all the relevant variables are going to be selected 
within TOj^g^x = sTOqjj^, steps. Exact values for and Wmax given below. 

Without loss of generality, we analyze the first step of the algorithm, i.e., we 
show that the first relevant variable is going to be selected within the first rn*,^^ 
steps. 

Assume that in the first m*^^ — 1 steps, there were no relevant variables 
selected. Assuming that the variable selected in the ml^^-th step is still an 
irrelevant one, we will arrive at a contradiction, which shows that at least one 
relevant variable has been selected in the first m*^^ steps. For any step fc, the 
reduction of the squared error is given as 

A(fc) RSS(fc - 1) - RSS(fc) = Y.^^jS'^^-^ " ^t,M^^Ml (17) 

t 

with Hj^) = X^^^X^I;)' 1 1X^^^11-2 and xj^ = (I„x„ - H,,^(.))Xt,,. We are 

interested in the quantity X^fc^i" when all the selected variables fk (see 
Algorithm [J) belong to [p]\M^,. 

In what follows, we will derive a lower bound for A(fc). We perform our 
analysis on the event 

£ = {min min A„iin(S>t) > (?!)min/2} 

te[T]MC[p],\M\<m* 

n{max max Amax(SA4) < 20,„ax}- 

I ^ te[T]MC[p],\M\<m*^^^ 
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(19) 



From the definition of fk , we liave 

A(fc) > max W^ili^n.n ~ H,,^(.,)y4||2 

> max (y^\\Il\''hl„xn -lit M('''>)'^t,M,f^t.Mj\l 

^ X/ (Inxn - Hj _vi(fc))et||2^ 

t 

— (Inxn — Hj A^(fc))Xf 7^,/3t^J|2 

— max ||h|'1- (I„xn — _;^(fc))et||2 
= (I) ill)- 

We deal with these two terms separately. Let = I„xn — ^t.,M denote the 

projection matrix. We have that the first term (/) is lower bounded by 

E( ^) -L 1 1 2 



= max J2 l|x|5||2-'|xg'H,^^<.,X,,^./3,,^j2 

* (20) 

> min {||Xg||2-n max ^ IX^'h^^^^X*,^./?*,^ 



t 



- ^tem'^eM, ll^*'^"'} ' max ^|X;^^.H^^,,,Xt,A^.A^A^j2^ 



where the last inequality follows from the fact that ||Xtj||2 > ||x|*j^||2 and 
xj'l- H;^^(fc) = Xj -H^^(fc). A simple calculation shows that 



t 



t jGM, 



(21) 



< ll/3||2,i max ^^(Xt,,H^-^^,„Xt,^,.A,.M.)^. 
Plugging (PT|) back into (PH)) . the following lower bound is achieved 

,,2t _i (St im-^^(.)Xt,^. A,>tjii)^ 

(7)>-^ max X(,U^ '■ — 



(22) 
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On the event £, 'niayLt£Yr].jeM, ll^tjlli ^ 2n(/)max- Since we have assumed that 
no additional relevant predictors have been selected by the procedure, it holds 
that % M^^\ This leads to 

y2 \\^t^MW^t,M,f3t,M.\\2 > 2"^n0nii„ min V (23) 

t tG[T] 

on the event 8. Using the Cauchy-Schwarz inequahty, ||B||^^ > s^^T^^C^^. 
Plugging back into 1^^, we have that 



^2 



(/) > 2-30^.^0-i^C^^,s-iT-i( mm ^ f^l^, 

(24) 



telT] 



Next, we deal with the second term in (fT9l). Recah that x|^^ = (,)Xtj, 



so that ||x|'|j- > 2 ^n^min, on the event £. We have 

-tib 



'^W^i'^j 0-nxn - H(^^(fc))et 

t 

= ^||X«ir2(X;,,.H^^,.,eO^ (25) 
t 

< max max ^(Xj -H.-^^eO^ 

Under the conditions of the theorem, XJ^H^^et is normally distributed with 
mean and variance ||H^^Xtj|||. Furthermore, 

max max max HH^'^A^Xt < 2n(/)inax- (26) 

J&M, \M\<m^^^telT] 

Plugging back in (|25|) . we have 

(//) < 2^(f>~l^(f>^^^ max max Xt: (27) 

where Xt denotes a chi-squared random variable with T degrees of freedom. 
The total number of possibilities for j G Ai* and \A4\ < ?7i*„ax bounded by 
p^max+s^ Using Lemma [SJ with e = T{m^^^ + 2) logp and applying the union 
bound, we obtain 

(//) < 230^[>„.axr(m*_ + 2) logp ^^^^ 
with probability at least 

1 _ p™.-„x+2 exp ( -2r(m;;,a, + 2) log(p) (1-2. 



2(r<ax + 2)log(p) 



(29) 
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Going back to (|19p . we have the fohowing 



telT] 



-1 * 

777 

max 



>2 "^^miii^max^^^^s "^'^^^ 



(30) 



mm 
1-3 i2 



4^maxCpTl ^ ^max 



>2 '^0niin<^max^/3^^s 

X (1 - 720-f„</.LxC|Cpac^2^*=+2*-"+^''-VC). 

Since the bound in (pHl) holds uniformly for fc G {1, . . . , ml^^}, we have that 
r>^-'T-'Ete[T] WytWl > n-'T-^ETlr Mk)- Setting 



[2 



4^-2 ^ 
min' 



and recalling that m* 



sTO*,j(,, the lower bound becomes 



te[T] 



|yt||2 > 2(1 -Cn 



3i5a+4i5„i„+i5„-l\ 



(31) 



(32) 



for a positive constant C independent of p, n, s and T. Under the conditions of 
the theorem, the right side of is bounded below by 2. We have arrived at 
a contradiction, since under the assumptions Var(yf^i) = 1 and by the weak law 
of large numbers, X]te[T] llytlli — ^ 1 in probability. Therefore, at least 

one relevant variable will be selected in m*^^ steps. 

To complete the proof, we lower bound the probability in psp and the prob- 
ability of the event £. Plugging in the value for mj^ax' the probability in (pSj) 
can be lower bounded by 1 — exp(— C(2T — i)rt253+2(5,„i„+5p-j some positive 
constant C. The probability of the event £ is lower bounded, using Lemma [3] 
together with the union bound, as 1 — Ci exp(— C2 ;„"x|iogp log'r} )i some pos- 
itive constants Ci and C2 ■ Both of these probabilities converge to 1 under the 
conditions of the theorem. 



6.2 Proof of Theorem H 

To prove the theorem, we use the same strategy as in IWang C2OO9I ). From 
TheoremlH we have that P[3fc € {0, . . . , n - 1} : C A^C^)] 1, so fc^in := 
ininfee{o,...,ri-i}{^ : M^, C A^C")} is well defined and 
defined in ([T3l) . We show that 

P[ min (BIC(Al('=)) - BIC(M^''+^^)) > 0] ^ 1, (33) 

fee{0,...,*;,„i„-l} 
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so that P[s < fcmin] — as « — ^ oo. We proceed by lower bounding the difference 
in the BIC scores as 



BIC(...«) - BIC(...-.., - ,0, ( Jg^) - i2SW±^ ^^^^ 



^ + RSS(Af(^-+i)) ^ j - 

where we have assumed p > n. Define the event A := {n^^T^^ 12telT] llytlli — 
2}. Note that RSS(A^('=+i)) < J2t&[T] llxtlli so on the event A the difference 
in the BIC scores is lower bounded as 

log(l + 2n-^T-'^A{k)) - 3n-^ log(p), (35) 

where A{k) is defined in ([T7|. Using the fact that log(l + a;) > min(log(2), 2"^x) 
and the lower bound from ([5(1)) . we have 

BICCXC^)) - BIC(X('=+i)) > min(log2,Cn~*=-2<5„„„) _ 3n-^\ogp, (36) 

for some positive constant C. It is easy to check that log 2 — 3n^^ logp > and 
Cri~''=~2*"'" — 3n~^ logp > under the conditions of the theorem. The lower 
bound in is uniform for k S {0, . . . , fcmin}, so the proof is complete if we 
show that P[^] — >■ 1. But this easily follows from the tail bounds on the central 
chi-squared random variable given in Lemma |4l 

6.3 Collection of known results 

In what follows, Ci, C2, . . . will denote arbitrary positive constants. 

The following result on the minimum e igenvalue of sub-matrices of the co- 
variance matrix S is quite standard (e.g. Zhou et al. (20091) . Wang C2009I) or 
Bickel et al. (20091) ). 



Lemma 3. Let x ~ Af{0, S) and S = n ^ X^iLi ^i^i empirical estimate 

from n independent realizations o/x. Denote S = [oab] cind S — [aab]- Assume 

0min < Amin(S) < Aniax(S) < 0max • Then 

P[ max Amax(SA4) > 20,„ax] < Ci exp(-C2^ + slogp) (37) 



Tl 

min Amin(SA4) < 0mi„/2] < C3exp(-C4^ + slogp). (38) 

MC[p],\M\<s S^ 



The following tail bounds for the chi-squared distribution are taken from lLaurent and Massart (20001) 



Lemma 4. Let Xn central chi-squared r.v. with n degrees of freedom. For 
any positive e, 

^xl >n + 2V^ + 2e] < exp(-e) (39) 
P[X^ <e-2V^] <exp(-e). (40) 
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We also make use of the result taken from Obozinski et al. (20091) ■ which 
bounds the maximum of a collection of chi-squared random variables. 

Lemma 5. Let Xi, . . . ,Xm he i.i.d. central chi-squared r.v. with n degrees of 
freedom. Then for any e > n, 

P[maxXi > 2e] < mexp(-e(l - 2./-)). (41) 
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6.4 Tables with simulation results 





Simulation 1: 


(n,p,s,T) 


= (500, 20000, 18, 500), Tnon-zoro 


— 500 








Method name 




Correct zeros Incorrect zeros 


ywt — o 


1^1 


l|B-B||i 












SNR = 15 












SIS-ALASSO 


100.0 


100.0 


0.0 


10.0 


20.2 


- 


- 


■i-l 

o 


ISIS-ALASSO 


100.0 


100.0 


0.0 


18.0 


19.6 


- 


- 


Uniol 
Sudd 


OMP 


100.0 


100.0 


0.0 


0.0 


23.9 


- 


- 


b-UiVlJr 


iUU.U 


100.0 


0.0 


1 nn n 
IUU.U 


18.0 


- 


- 




S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


18.0 








SIS-ALASSO 


0.0 


100.0 


0.7 


0.0 


8940.5 


0.97 


0.93 


O Oh 


ISIS-ALASSO 


100.0 


100.0 


0.0 


18.0 


9001.6 


0.33 


0.93 




OMP 


100.0 


100.0 


0.0 


0.0 


9005.9 


0.20 


0.93 


Hod 




1 nn n 
iUU.U 


100.0 


0.0 


1 nn n 
iUU.U 


9000.0 


0.20 


0.93 










SNR = 10 












SIS-ALASSO 


100.0 


100.0 


0.0 


0.0 


25.3 


- 


- 


r( O 


ISIS-ALASSO 


100.0 


100.0 


0.0 


0.0 


25.7 


- 


- 


Unio] 
Sudd 


OMP 


100.0 


100.0 


0.0 


0.0 


23.9 


- 


- 


b-UiVlJr 


IUU.U 


100.0 


0.0 


1 nn n 
IUU.U 


18.0 


- 


- 




S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


18.0 








SIS-ALASSO 


0.0 


100.0 


1.6 


0.0 


8861.0 


2.06 


0.89 


s 

o ft 


ISIS-ALASSO 


100.0 


100.0 


0.0 


0.0 


9007.7 


0.65 


0.90 


03 fx 

Hm 


OMP 


100.0 

00. U 


100.0 
100.0 


0.0 
0.1 


0.0 

OO.U 


9005.9 
8987.4 


0.31 
0.41 


0.91 
0.90 










SNR = 5 












SIS-ALASSO 


100.0 


100.0 


0.0 


64.0 


18.4 


- 


- 


r< O 


ISIS-ALASSO 


100.0 


100.0 


0.0 


57.0 


18.6 


- 


- 


Uniol 
Sudd 


OMP 


100.0 


100.0 


0.0 


0.0 


24.0 


- 


- 


o-KJNLr 


iUU.U 


100.0 


0.0 


1 nn n 
IUU.U 


18.0 


- 


- 




S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


18.0 








SIS-ALASSO 


0.0 


100.0 


92.8 


0.0 


645.8 


74.61 


0.06 


^ S 

o ft 


ISIS-ALASSO 


0.0 


100.0 


90.9 


0.0 


822.2 


73.06 


0.07 


ffl ft 
X 3 


OMP 
S-OMP-ALASSO 


100.0 
0.0 


100.0 
100.0 


0.0 
70.3 


0.0 
0.0 


9006.0 
2668.9 


0.61 
56.65 


0.83 
0.24 










SNR = 1 












SIS-ALASSO 


0.0 


100.0 


99.9 


0.0 


0.0 






4^ 

;h 

H O 


ISIS-ALASSO 


0.0 


100.0 


100.0 


0.0 


0.0 






Uniol 
Sudd 


OMP 


100.0 


100.0 


0.0 


0.0 


25.9 






S-OMP 


0.0 


100.0 


94.4 


0.0 


1.0 








S-OMP-ALASSO 


0.0 


100.0 


99.0 


0.0 


0.2 








SIS-ALASSO 


0.0 


100.0 


100.0 


0.0 


0.0 


80.27 


-0.00 




ISIS-ALASSO 


0.0 


100.0 


100.0 


0.0 


0.0 


80.27 


-0.00 


cS CL 

>^ ? 


OMP 


0.0 


100.0 


86.5 


0.0 


1222.8 


71.40 


0.05 




S-OMP-ALASSO 


0.0 


100.0 


100.0 


0.0 


0.2 


80.27 


-0.00 
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Simulation 1: 


{n,p,s,T) 


= (500, 20000, 18, 500), Tnon-zero 


= 300 








A/l6tlioci iiaiii6 


M, C S 


Correct zeros Incorrect zeros 


M* = S 




l|B - Blli 












SNR = 


15 












AT AS^D 


1 nn n 

iUU.U 


100.0 




0.0 


Q7 n 


18.0 




~ 


n O 




1 nn n 


100.0 




0.0 


yo.u 


18.0 






Unio] 
Supp 


OMP 


1 nn n 


100.0 




0.0 


n n 


23.0 






o OA/TP 


1 nn n 


100.0 




0.0 


1 nn n 

iUU.U 


18.0 








S-OMP-ALASSO 


100.0 


100.0 




0.0 


100.0 


18.0 








QTQ AT A^mO 




100.0 




0.0 


n 

oo.u 


5399.3 


0.10 


0.93 




TCTQ AT A'^'^^ 


1 nn n 


100.0 




0.0 


QR n 
yo.u 


5400.0 


0.09 


0.93 




OMP 


100.0 


100.0 




0.0 


0.0 


5405.0 


0.07 


0.93 


Hco 


S-OMP-ALASSO 


100.0 


100.0 




0.0 


100.0 


5400.0 


0.07 


0.93 










SNR = 


10 












GTC AT AQQO 


1 nn n 


100.0 




0.0 


S9 n 


18.2 






O 


AT AQQO 


1 nn n 


100.0 




0.0 


Q1 n 

yi.u 


18.1 






Uniol 
Sudd 


OA/TP 


1 nn n 


100.0 




0.0 


n n 

u.u 


23.0 






q OA/TP 

o- WiVlX 


1 nn n 


100.0 




0.0 


1 nn n 

iUU.U 


18.0 








S-OMP-ALASSO 


100.0 


100.0 




0.0 


100.0 


18.0 








^T^ AT A^t^D 




100.0 




0.0 


oo.u 


5399.2 


0.18 


0.90 


-^^ s 

o ft 


T<5TC! AT A'^cin 


1 nn n 

IUU.U 


100.0 




0.0 


Q1 n 
yi.u 


5400.1 


0.16 


0.90 


c6 ft 


OMP 
S-OMl'-ALASSO 


ino.o 

100.0 


iUU.U 
100.0 




u.u 

0.0 


0.0 

100.0 


5405.0 
5100.0 


O.U 
O.ii 


0.90 
0.90 










SNR = 


5 












QTQ AT AQQO 


1 nn n 

IUU.U 


100.0 




0.0 


^ 
o.U 


21.1 






r! O 


TQTQ AT A'^'^^ 


1 on n 

IUU.U 


100.0 




0.0 


f\ 
u.u 


20.8 






Unio] 
Supp 


DMP 


IUU.U 


100.0 




0.0 



u.u 


23.0 






Q ni\/rp 


1 nn n 

IUU.U 


100.0 




0.0 


1 nn n 

IUU.U 


18.0 








S-OMP-ALASSO 


100.0 


100.0 




0.0 


100.0 


18.0 








QTQ AT Ac;c!n 


9A n 


100.0 




0.0 


1 n 

l.U 


5400.9 


0.61 


0.82 


^ g 
O ft 


TtJTC; AT ACJCin 
ioio-AijAooW 


QQ n 


100.0 




0.0 


R n 

D.U 


5402.8 


0.52 


0.82 


Hco 


DA/TP 

S-OMP-ALASSO 


IUU.U 

100.0 


1 nn n 
iUU.U 

100.0 




n n 
u.u 

0.0 


n n 
u.u 

100.0 


5405.0 
5400.0 


0.22 
0.23 


0.82 
0.82 










SNR = 


1 












SIS-ALASSO 


0.0 


100.0 




97.9 


0.0 


0.4 






n O 


ISIS-ALASSO 


0.0 


100.0 




97.9 


0.0 


0.4 






Unio] 
Sudd 


OMP 


100.0 


100.0 




0.0 


0.0 


25.9 






S-OMP 


0.0 


100.0 




94.4 


0.0 


1.0 








S-OMP-ALASSO 


0.0 


100.0 




94.4 


0.0 


1.0 








SIS-ALASSO 


0.0 


100.0 




100.0 


0.0 


0.4 


48.16 


-0.00 


^ s 

o ft 


ISIS-ALASSO 


0.0 


100.0 




100.0 


0.0 


0.4 


48.16 


-0.00 


ft 


OMP 


0.0 


100.0 




10.2 


0.0 


4858.1 


5.76 


0.43 




S-OMP-ALASSO 


0.0 


100.0 




99.9 


0.0 


6.1 


48.12 


-0.00 
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Simulation 1: 


{n,p,s,T) = 


= (500, 20000, 18, 500), Tnon-zcro = 


100 








A/l6tlioci iiaiii6 


M, C S 


Correct zeros Incorrect zeros 


M* = S 




l|B - Bill 












SNR = 


15 












AT AS^D 


1 nn n 

iUU.U 


100.0 




0.0 


1 nn n 


18.0 






n O 




1 nn n 


100.0 




0.0 


1 nn n 

iUU.U 


18.0 






Unio] 
Supp 


OMP 


1 nn n 


99.9 




0.0 


n n 

u.u 


28.8 






o OA/TP 


1 nn n 


100.0 




0.0 


1 nn n 

iUU.U 


18.0 








S-OMP-ALASSO 


100.0 


100.0 




0.0 


100.0 


18.0 








QTQ AT A^mO 


1 nn n 

IUU.U 


100.0 




0.0 


1 nn n 

iUU.U 


1800.0 


0.01 


0.91 




TCTQ AT A'^'^^ 


1 nn n 

iUU.U 


100.0 




0.0 


1 nn n 

iUU.U 


1800.0 


0.01 


0.91 




OMP 


100.0 


100.0 




0.0 


0.0 


1810.8 


0.01 


0.91 


Hco 


S-OMP-ALASSO 


100.0 


100.0 




0.0 


100.0 


1800.0 


0.01 


0.91 










SNR = 


10 












GTC AT AQQO 


1 nn n 


100.0 




0.0 


1 nn n 

iUU.U 


18.0 






O 


AT AQQO 


1 nn n 

iUU.U 


100.0 




0.0 


1 nn n 

iUU.U 


18.0 






Uniol 
Sudd 


OA/TP 


1 nn n 


99.9 




0.0 


n n 

U.U 


28.8 






q OA/TP 

o- WiVlX 


1 nn n 

iUU.U 


100.0 




0.0 


1 nn n 

iUU.U 


18.0 








S-OMP-ALASSO 


100.0 


100.0 




0.0 


100.0 


18.0 








^T^ AT A^t^D 


1 no n 

IUU.U 


100.0 




0.0 


1 00 n 

IUU.U 


1800.0 


0.01 


0.88 


-^^ s 

o ft 


T<5TC! AT A'^cin 


1 nn n 

IUU.U 


100.0 




0.0 


1 00 
IUU.U 


1800.0 


0.01 


0.88 


c6 ft 


OMP 
S-OMl'-ALASSO 


ino.o 

100.0 


iUU.U 
100.0 




u.u 

0.0 


0.0 

100.0 


1810.8 
1800.0 


0.01 
O.Oi 


0.88 
0.88 










SNR = 


5 












QTQ AT AQQO 


1 nn n 

IUU.U 


100.0 




0.0 


1 nn n 

iUU.U 


18.0 






r! O 


TQTQ AT A'^'^^ 


1 on n 

IUU.U 


100.0 




0.0 


1 00 n 

IUU.U 


18.0 






Unio] 
Supp 


DMP 


IUU.U 


99.9 




0.0 



u.u 


28.8 






Q ni\/rp 


1 nn n 

IUU.U 


100.0 




0.0 


1 00 n 

±uu.u 


18.0 








S-OMP-ALASSO 


100.0 


100.0 




0.0 


100.0 


18.0 








QTQ AT Ac;c!n 


1 nn n 

iUU.U 


100.0 




0.0 


1 nn n 

iUU.U 


1800.0 


0.04 


0.79 


^ g 
O ft 


TtJTC; AT ACJCin 
ioio-AijAooW 


1 nn n 

iUU.U 


100.0 




0.0 


1 nn n 

iUU.U 


1800.0 


0.03 


0.79 


Hco 


DA/TP 

S-OMP-ALASSO 


IUU.U 

100.0 


1 nn n 
iUU.U 

100.0 




n n 
u.u 

0.0 


n n 
u.u 

100.0 


1810.8 
1800.0 


0.03 
0.02 


0.79 
0.79 










SNR = 


1 












SIS-ALASSO 


100.0 


100.0 




0.0 


19.0 


19.6 






n O 


ISIS-ALASSO 


100.0 


100.0 




0.0 


35.0 


19.0 






Unio] 
Sudd 


OMP 


100.0 


99.9 




0.0 


0.0 


28.8 






S-OMP 


0.0 


100.0 




94.4 


0.0 


1.0 








S-OMP-ALASSO 


0.0 


100.0 




94.4 


0.0 


1.0 








SIS-ALASSO 


59.0 


100.0 




0.0 


10.0 


1800.9 


0.74 


0.45 


^ s 

o ft 


ISIS-ALASSO 


89.0 


100.0 




0.0 


32.0 


1800.8 


0.63 


0.45 


ft 


OMP 


100.0 


100.0 




0.0 


0.0 


1810.8 


0.13 


0.47 




S-OMP-ALASSO 


0.0 


100.0 




95.3 


0.0 


84.6 


15.31 


0.02 
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Simulation 2. a: 


{n,p,s,T) 


= (200, 5000, 10, 500), Tnon-zero 


= 400 








A/l6tlioci iiaiii6 


M, C S 


Correct zeros Incorrect zeros 


M* = S 




l|B - Blli 












SNR = 10 












AT AS^D 


1 nn n 

lUU.U 


100.0 


0.0 


"iQ n 


10.9 






n O 




1 nn n 


100.0 


0.0 


IZ.U 


12.2 






Unio] 
Supp 


OMP 


1 nn n 


99.8 


0.0 


n n 


21.6 






o OA/TP 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 








QTQ AT A^mO 


n n 


100.0 


6.4 


n n 

u.u 


3746.6 


3.58 


0.85 




TCTQ AT A'^'^^ 


41 n 


100.0 


0.2 


^ n 

o.u 


3992.8 


0.53 


0.90 




OMP 


100.0 


100.0 


0.0 


0.0 


4011.7 


0.22 


0.90 


Hco 


S-OMP-ALASSO 


99.0 


100.0 


0.0 


98.0 


3999.9 


0.22 


0.90 










SNR = 5 












GTC AT AQQO 


1 nn n 


100.0 


0.0 


4^ n 


11.0 






O 


AT AQQO 


1 nn n 


100.0 


0.0 


V7 n 

O 1 .u 


10.9 






Uniol 
Sudd 


OA/TP 


1 nn n 


99.8 


0.0 


n n 

u.u 


22.2 






q OA/TP 

o- WiVlX 


1 nn n 


100.0 


0.0 


1 nn n 

lUU.U 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 








^T^ AT A^t^D 

OlO r\.l-;r\.00 W 


u.u 


100.0 


65.9 


u.u 


1363.5 


33.32 


0.30 


-^^ s 

o ft 


T<5TC! AT A'^cin 
1 o 1 o - rl Ij r\ o o v-/ 


n n 
u.u 


100.0 


63.1 


u.u 


1477.0 


31.89 


0.33 


c6 ft 


OMP 
S-OMl'-ALASSO 


100.0 
0.0 


1 nn n 

iUU.U 

100.0 


U . U 
18.0 


0.0 

0.0 


4012.2 
2081.5 


0.45 
21.1!) 


0.82 
0. 10 










SNR = 1 












SIS-ALASSO 


0.0 


100.0 


98.2 


0.0 


0.2 






r! O 


ISIS-ALASSO 


0.0 


100.0 


98.7 


0.0 


0.1 






Unio] 
Supp 


OMP 


100.0 


99.5 


0.0 


0.0 


35.2 






S-OMP 


0.0 


100.0 


90.0 


0.0 


1.0 








S-OMP-ALASSO 


0.0 


100.0 


95.4 


0.0 


0.5 








SIS-ALASSO 


0.0 


100.0 


100.0 


0.0 


0.2 


49.94 


-0.00 


^ g 
O ft 


ISIS-ALASSO 


0.0 


100.0 


100.0 


0.0 


0.1 


49.94 


-0.00 


Hco 


OMP 
S-OMP-ALASSO 


0.0 
0.0 


100.0 
100.0 


76.5 
100.0 


0.0 
0.0 


964.4 
0.8 


40.05 
49.94 


0.09 
-0.00 
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Simulation 2. a: 


{n,p,s,T) 


= (200, 5000, 10, 500), Tnon-zero 


= 250 








A/l6tlioci iiaiii6 


M, C S 


Correct zeros Incorrect zeros 


M* = S 




l|B - Blli 












SNR = 10 












AT AS^D 


1 nn n 

lUU.U 


100.0 


0.0 


QQ n 


10.0 




~ 


n O 




1 nn n 


100.0 


0.0 


yo.u 


10.0 






Unio] 
Supp 


OMP 


1 nn n 


99.8 


0.0 


n n 


19.9 






o OA/TP 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 








QTQ AT A^mO 


99 n 


100.0 


0.2 


99 n 


2495.4 


0.19 


0.89 




TCTQ AT A'^'^^ 


1 nn n 


100.0 


0.0 


QR n 
yo.u 


2500.0 


0.12 


0.89 




OMP 


100.0 


100.0 


0.0 


0.0 


2509.9 


0.09 


0.90 


Hco 


S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


2500.0 


0.08 


0.90 










SNR = 5 












GTC AT AQQO 


1 nn n 


100.0 


0.0 


44 n 


10.8 






O 


AT AQQO 


1 nn n 


100.0 


0.0 


4fi n 


10.8 






Uniol 
Sudd 


OA/TP 


1 nn n 


99.8 


0.0 


n n 

u.u 


19.9 






q OA/TP 

o- WiVlX 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 








^T^ AT A^t^D 

OlO ril^riOO W 




100.0 


0.9 


u.u 


2479.5 


0.69 


0.80 


-^^ s 

o ft 


T<5TC! AT A'^cin 
1 o 1 o - rl Ij r\ o o v-/ 


DZ.U 


100.0 


0.2 


9Q n 
zy.u 


2496.7 


0.43 


0.81 


c6 ft 


OMP 
S-OMl'-ALASSO 


100.0 

95.0 


1 nn n 
100.0 


U . U 
0.0 


0.0 

95.0 


2509.9 
2199.G 


0.18 
0.i« 


0.81 
O.Si 










SNR = 1 












SIS-ALASSO 


0.0 


100.0 


65.3 


0.0 


3.5 






r! O 


ISIS-ALASSO 


0.0 


100.0 


61.3 


0.0 


3.9 






Unio] 
Supp 


OMP 


100.0 


99.7 


0.0 


0.0 


24.7 






S-OMP 


0.0 


100.0 


90.0 


0.0 


1.0 








S-OMP-ALASSO 


0.0 


100.0 


90.0 


0.0 


1.0 








SIS-ALASSO 


0.0 


100.0 


99.8 


0.0 


4.6 


31.16 


-0.00 


^ g 
O ft 


ISIS-ALASSO 


0.0 


100.0 


99.8 


0.0 


5.2 


31.15 


-0.00 


Hco 


OMP 
S-OMP-ALASSO 


0.0 
0.0 


100.0 
100.0 


17.2 
99.6 


0.0 
0.0 


2083.7 
10.4 


6.09 
31.11 


0.39 
-0.00 
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Simulation 2. a: 


(n,p, s,T) 


= (200, 5000, 10, 500), Tnon^zero 


= 100 








A/l6tlioci iiam6 


M, C S 


Correct zeros Incorrect zeros 


M» = S 




l|B - Bill 












SNR = 10 












AT AS^D 


1 nn n 

lUU.U 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 






n O 


mjC! AT A^cin 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 






Unio] 
Supp 


OMP 


1 nn n 


98.8 


0.0 


n n 

u.u 


69.8 






o OA/TP 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 








QTQ AT A^mO 


n 


100.0 


0.0 


Qs n 


1000.0 


0.02 


0.80 




TCTQ AT A'^'^^ 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


1000.0 


0.01 


0.80 




OMP 


100.0 


100.0 


0.0 


0.0 


1060.2 


0.02 


0.79 


Hco 


S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


1000.0 


0.01 


0.80 










SNR = 5 












GTC AT AQQO 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 






O 


AT AQQO 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 






Uniol 
Sudd 


OA/TP 


1 nn n 


98.8 


0.0 


n n 

u.u 


69.8 






q OA/TP 

o- WiVlX 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 








^T^ AT A^t^D 

OlO r\.l-;r\.00 W 


yo.u 


100.0 


0.0 


aO.U 


1000.0 


0.04 


0.73 


-^^ s 

o ft 


T<5TC! AT A'^cin 
1 o 1 o - rl Ij r\ o o v-/ 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


1000.0 


0.04 


0.73 


c6 ft 


OMP 
S-OMl'-ALASSO 


100.0 
iOO.O 


1 nn n 
100.0 


n n 
0.0 


0.0 

iOO.O 


1060.2 
iOOO.O 


0.05 
0.0.3 


0.72 
0.73 










SNR = 1 












SIS-ALASSO 


100.0 


100.0 


0.0 


61.0 


10.6 






r! O 


ISIS-ALASSO 


100.0 


100.0 


0.0 


60.0 


10.5 






Unio] 
Supp 


OMP 


100.0 


98.8 


0.0 


0.0 


69.8 






S-OMP 


0.0 


100.0 


90.0 


0.0 


1.0 








S-OMP-ALASSO 


0.0 


100.0 


90.0 


0.0 


1.0 








SIS-ALASSO 


0.0 


100.0 


12.7 


0.0 


873.9 


2.23 


0.37 


^ g 
O ft 


ISIS-ALASSO 


0.0 


100.0 


9.8 


0.0 


902.8 


1.79 


0.38 


Hco 


OMP 
S-OMP-ALASSO 


100.0 
0.0 


100.0 
100.0 


0.0 
93.3 


0.0 
0.0 


1060.2 
67.4 


0.25 
11.66 


0.42 
0.03 
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Simulation 2.b: 


{n,p,s,T) 


= (200, 5000, 10, 750), T^on-zero 


= 600 








A/l6tlioci iiaiii6 


M, C S 


Correct zeros Incorrect zeros 


M* = S 




l|B - Blli 












SNR = 10 












AT AS^D 


1 nn n 

lUU.U 


100.0 


0.0 


9^ n 


11.3 




~ 


n O 




1 nn n 


99.9 


0.0 


^ n 


13.3 






Unio] 
Supp 


OMP 


1 nn n 


99.7 


0.0 


n n 

u.u 


26.6 






o OA/TP 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 








QTQ AT A^mO 


n n 


100.0 


6.9 


n n 

u.u 


5585.0 


3.87 


0.84 




TCTQ AT A'^'^^ 


9Q n 


100.0 


0.3 


4 n 

^.u 


5986.6 


0.56 


0.90 




OMP 


100.0 


100.0 


0.0 


0.0 


6016.7 


0.22 


0.90 


Hco 


S-OMP-ALASSO 


91.0 


100.0 


0.0 


91.0 


5999.1 


0.23 


0.90 










SNR = 5 












GTC AT AQQO 


1 nn n 


100.0 


0.0 


97 n 

^ i .u 


11.4 






O 


AT AQQO 


1 nn n 


100.0 


0.0 


98 n 

zo.u 


11.3 






Uniol 
Sudd 


OA/TP 


1 nn n 


99.7 


0.0 


n n 


27.3 






q OA/TP 

o- WiVlX 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 








^T^ AT A^t^D 

OlO r\.l-;r\.00 W 


u.u 


100.0 


66.5 


u.u 


2011.9 


33.60 


0.30 


-^^ s 

o ft 


T<5TC! AT A'^cin 
1 o 1 o - rl Ij r\ o o v-/ 


n n 
u.u 


100.0 


63.6 


u.u 


2185.7 


32.14 


0.32 


c6 ft 


OMP 
S-OMl'-ALASSO 


100.0 
0.0 


1 nn n 

iUU.U 

100.0 


u.u 

18.:.! 


0.0 

0.0 


6017.5 
:il01.1 


0.45 
2l.:il 


0.82 
0. 15 










SNR = 1 












SIS-ALASSO 


0.0 


100.0 


97.8 


0.0 


0.2 






r! O 


ISIS-ALASSO 


0.0 


100.0 


98.2 


0.0 


0.2 






Unio] 
Supp 


OMP 


100.0 


99.2 


0.0 


0.0 


47.6 






S-OMP 


0.0 


100.0 


90.0 


0.0 


1.0 








S-OMP-ALASSO 


0.0 


100.0 


94.7 


0.0 


0.5 








SIS-ALASSO 


0.0 


100.0 


100.0 


0.0 


0.2 


49.94 


-0.01 


^ g 
O ft 


ISIS-ALASSO 


0.0 


100.0 


100.0 


0.0 


0.2 


49.94 


-0.01 


Hco 


OMP 
S-OMP-ALASSO 


0.0 
0.0 


100.0 
100.0 


76.7 
100.0 


0.0 
0.0 


1436.7 
1.0 


40.13 
49.94 


0.09 
-0.01 
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Simulation 2.b: 


{n,p,s,T) 


= (200, 5000, 10, 750), T^on-zero 


= 375 








A/l6tlioci iiaiii6 


M, C S 


Correct zeros Incorrect zeros 


M* = S 




l|B - Blli 












SNR = 10 












AT AS^D 


1 nn n 

lUU.U 


100.0 


0.0 


QQ n 


10.0 




~ 


n O 




1 nn n 


100.0 


0.0 


yo.u 


10.1 






Unio] 
Supp 


OMP 


1 nn n 


99.7 


0.0 


n n 


24.7 






o OA/TP 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 








QTQ AT A^mO 


1 fi n 


100.0 


0.2 


1 fi n 


3741.3 


0.21 


0.89 




TCTQ AT A'^'^^ 


1 nn n 


100.0 


0.0 


Q'^ n 

yo.u 


3750.1 


0.12 


0.89 




OMP 


100.0 


100.0 


0.0 


0.0 


3764.8 


0.09 


0.89 


Hco 


S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


3750.0 


0.09 


0.89 










SNR = 5 












GTC AT AQQO 


1 nn n 


100.0 


0.0 


41 n 

^l.U 


10.9 






O 


AT AQQO 


1 nn n 


100.0 


0.0 


9^ n 


11.4 






Uniol 
Sudd 


OA/TP 


1 nn n 


99.7 


0.0 


n n 

u.u 


24.7 






q OA/TP 

o- WiVlX 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 








^T^ AT A^t^D 

OlO r\.l-;r\.00 W 


u.u 


100.0 


1.0 


O.KJ 


3713.5 


0.73 


0.80 


-^^ s 

o ft 


T<5TC! AT A'^cin 
1 o 1 o - rl Ij r\ o o v-/ 


OO.U 


100.0 


0.2 


lo.U 


3744.9 


0.43 


0.80 


c6 ft 


OMP 
S-OMl'-ALASSO 


100.0 

iJi.O 


1 nn n 
100.0 


U . U 
0.0 


0.0 

9i.0 


3764.8 
3719.0 


0.18 
0.19 


0.81 
0.81 










SNR = 1 












SIS-ALASSO 


0.0 


100.0 


55.8 


0.0 


4.4 






r! O 


ISIS-ALASSO 


1.0 


100.0 


52.8 


1.0 


4.7 






Unio] 
Supp 


OMP 


100.0 


99.6 


0.0 


0.0 


32.0 






S-OMP 


0.0 


100.0 


90.0 


0.0 


1.0 








S-OMP-ALASSO 


0.0 


100.0 


90.0 


0.0 


1.0 








SIS-ALASSO 


0.0 


100.0 


99.8 


0.0 


6.6 


31.16 


-0.00 


^ g 
O ft 


ISIS-ALASSO 


0.0 


100.0 


99.8 


0.0 


7.3 


31.16 


-0.00 


Hco 


OMP 
S-OMP-ALASSO 


0.0 
0.0 


100.0 
100.0 


17.6 
99.6 


0.0 
0.0 


3111.8 
15.1 


6.21 
31.11 


0.39 
-0.00 
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Simulation 2.b: 


{n,p,s,T) 


= (200, 5000, 10, 750), Tnon-zero 


= 150 








A/l6tlioci iiam6 


M, C S 


Correct zeros Incorrect zeros 


M* = S 




l|B - Bill 












SNR = 10 












AT AS^D 


1 nn n 

lUU.U 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 






n O 


mjC! AT A^cin 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 






Unio] 
Supp 


OMP 


1 nn n 


98.0 


0.0 


n n 

u.u 


108.5 






o OA/TP 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 








QTQ AT A^mO 


n 


100.0 


0.0 


Qs n 


1500.0 


0.02 


0.79 




TCTQ AT A'^'^^ 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


1500.0 


0.02 


0.79 




OMP 


100.0 


100.0 


0.0 


0.0 


1599.5 


0.03 


0.78 


Hco 


S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


1500.0 


0.01 


0.79 










SNR = 5 












GTC AT AQQO 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 






O 


AT AQQO 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 






Uniol 
Sudd 


OA/TP 


1 nn n 


98.0 


0.0 


n n 

u.u 


108.5 






q OA/TP 

o- WiVlX 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 








^T^ AT A^t^D 

OlO ril^riOO W 


yo.u 


100.0 


0.0 


aO.U 


1500.0 


0.04 


0.72 


-^^ s 

o ft 


T<5TC! AT A'^cin 
1 o 1 o - rl Ij r\ o o v-/ 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


1500.0 


0.04 


0.72 


c6 ft 


OMP 
S-OMl'-ALASSO 


100.0 
100.0 


1 nn n 
100.0 


n n 
0.0 


0.0 

100.0 


1599.5 
1500.0 


0.05 
0.0.3 


0.71 
0.72 










SNR = 1 












SIS-ALASSO 


100.0 


100.0 


0.0 


46.0 


10.8 






r! O 


ISIS-ALASSO 


100.0 


100.0 


0.0 


42.0 


10.8 






Unio] 
Supp 


OMP 


100.0 


98.0 


0.0 


0.0 


108.5 






S-OMP 


0.0 


100.0 


90.0 


0.0 


1.0 








S-OMP-ALASSO 


0.0 


100.0 


90.0 


0.0 


1.0 








SIS-ALASSO 


0.0 


100.0 


12.1 


0.0 


1318.9 


2.16 


0.37 


^ g 
O ft 


ISIS-ALASSO 


0.0 


100.0 


9.4 


0.0 


1360.3 


1.74 


0.38 


Hco 


OMP 
S-OMP-ALASSO 


100.0 
0.0 


100.0 
100.0 


0.0 
93.4 


0.0 
0.0 


1599.5 
98.9 


0.26 
11.68 


0.42 
0.03 



34 





Simulation 2.c: 


{n,p,s,T) 


= (200,5000, 10, 1000), Tnon-zero 


= 800 








A/l6tlioci iiam6 


M, C S 


Correct zeros Incorrect zeros 


M* = S 




l|B - Blli 












SNR = 10 












AT AS^D 


1 nn n 

lUU.U 


100.0 


0.0 


91 n 

Zl .u 


11.7 




~ 


n O 


mjC! AT A^cin 


1 nn n 


99.9 


0.0 


^ n 


14.4 






Unio] 
Supp 


OMP 


1 nn n 


99.6 


0.0 


n n 

u.u 


32.0 






o OA/TP 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 








QTQ AT A^mO 


n n 


100.0 


7.7 


n n 

u.u 


7382.7 


4.26 


0.84 




TCTQ AT A'^'^^ 


1 7 n 

1 1 .u 


100.0 


0.4 


1 n 

l.U 


7976.0 


0.60 


0.90 




OMP 


100.0 


100.0 


0.0 


0.0 


8022.1 


0.22 


0.90 


Hco 


S-OMP-ALASSO 


86.0 


100.0 


0.0 


86.0 


7998.3 


0.23 


0.90 










SNR = 5 












GTC AT AQQO 


1 nn n 


100.0 


0.0 


1 4 n 

l^.U 


11.9 






O 


AT AQQO 


1 nn n 


100.0 


0.0 


1 7 n 

1 1 .u 


11.7 






Uniol 
Sudd 


OA/TP 


1 nn n 


99.5 


0.0 


n n 

u.u 


33.0 






q OA/TP 

o- WiVlX 


1 nn n 


100.0 


0.0 


1 nn n 

lUU.U 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 








^T^ AT A^t^D 

OlO r\.l-;r\.00 W 


u.u 


100.0 


65.5 


u.u 


2759.0 


33.13 


0.31 


-^^ s 

o ft 


T<5TC! AT A'^cin 
1 o 1 o - rl Ij r\ o o v-/ 


n n 
u.u 


100.0 


62.7 


u.u 


2984.0 


31.71 


0.33 


c6 ft 


OMP 
S-OMl'-ALASSO 


100.0 
0.0 


1 nn n 

iUU.U 

100.0 


U . U 

18. i 


0.0 

0.0 


8023.1 
1152.9 


0.45 
21.25 


0.82 
0. 10 










SNR = 1 












SIS-ALASSO 


0.0 


100.0 


97.6 


0.0 


0.2 






r! O 


ISIS-ALASSO 


0.0 


100.0 


97.3 


0.0 


0.3 






Unio] 
Supp 


OMP 


100.0 


99.0 


0.0 


0.0 


59.5 






S-OMP 


0.0 


100.0 


90.0 


0.0 


1.0 








S-OMP-ALASSO 


0.0 


100.0 


93.0 


0.0 


0.7 








SIS-ALASSO 


0.0 


100.0 


100.0 


0.0 


0.3 


49.94 


-0.01 


^ g 
O ft 


ISIS-ALASSO 


0.0 


100.0 


100.0 


0.0 


0.3 


49.94 


-0.01 


Hco 


OMP 
S-OMP-ALASSO 


0.0 
0.0 


100.0 
100.0 


76.4 
100.0 


0.0 
0.0 


1942.8 
1.8 


39.98 
49.94 


0.10 
-0.01 
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Simulation 2.c: 


{n,p,s,T) 


= (200,5000, 10, 1000), Tnon-zero 


= 500 








A/l6tlioci iiam6 


M, C S 


Correct zeros Incorrect zeros 


M* = S 




l|B - Blli 












SNR = 10 












AT AS^D 


1 nn n 

lUU.U 


100.0 


0.0 


SQ n 


10.1 




~ 


n O 


mjC! AT A^cin 


1 nn n 


100.0 


0.0 


yo.u 


10.1 






Unio] 
Supp 


OMP 


1 nn n 


99.6 


0.0 


n n 


29.1 






o OA/TP 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 








QTQ AT A^mO 


1 n 


100.0 


0.2 


1 n 


4990.2 


0.19 


0.89 




TCTQ AT A'^'^^ 


1 nn n 


100.0 


0.0 


n 


5000.1 


0.12 


0.89 




OMP 


100.0 


100.0 


0.0 


0.0 


5019.2 


0.09 


0.89 


Hco 


S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


5000.0 


0.09 


0.89 










SNR = 5 












GTC AT AQQO 


1 nn n 


100.0 


0.0 


97 n 

^ i .u 


11.4 






O 


AT AQQO 


1 nn n 


100.0 


0.0 


1 4 n 

l^.U 


11.6 






Uniol 
Sudd 


OA/TP 


1 nn n 


99.6 


0.0 


n n 

u.u 


29.1 






q OA/TP 

o- WiVlX 


1 nn n 


100.0 


0.0 


1 nn n 

lUU.U 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 








^T^ AT A^t^D 

OlO r\.l-;r\.00 W 


1 .u 


100.0 


0.8 


u.u 


4958.9 


0.69 


0.80 


-^^ s 

o ft 


T<5TC! AT A'^cin 
1 o 1 o - rl Ij r\ o o v-/ 


■^Q n 
oy .u 


100.0 


0.2 


inn 

lU.U 


4991.9 


0.44 


0.81 


c6 ft 


OMP 
S-OMl'-ALASSO 


100.0 

88.0 


1 nn n 
100.0 


U . U 
0.0 


0.0 

87.0 


5019.2 
1998.8 


0.18 
0.19 


0.81 
0.8i 










SNR = 1 












SIS-ALASSO 


0.0 


100.0 


46.3 


0.0 


5.4 






r! O 


ISIS-ALASSO 


1.0 


100.0 


42.8 


1.0 


5.7 






Unio] 
Supp 


OMP 


100.0 


99.4 


0.0 


0.0 


38.9 






S-OMP 


0.0 


100.0 


90.0 


0.0 


1.0 








S-OMP-ALASSO 


0.0 


100.0 


90.0 


0.0 


1.0 








SIS-ALASSO 


0.0 


100.0 


99.8 


0.0 


8.6 


31.16 


-0.00 


^ g 
O ft 


ISIS-ALASSO 


0.0 


100.0 


99.8 


0.0 


9.6 


31.16 


-0.00 


Hco 


OMP 
S-OMP-ALASSO 


0.0 
0.0 


100.0 
100.0 


17.5 
99.6 


0.0 
0.0 


4155.6 
20.1 


6.16 
31.11 


0.39 
-0.00 
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Simulation 2.c: 


{n,p,s,T) 


= (200, 5000, 10, 1000), Tnon-zero 


= 200 








A/l6tlioci iiaiii6 


M, C S 


Correct zeros Incorrect zeros 


M* = S 




l|B - Bill 












SNR = 10 












AT AS^D 


1 nn n 

lUU.U 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 






n O 




1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 






Unio] 
Supp 


OMP 


1 nn n 


97.4 


0.0 


n n 

u.u 


139.6 






o OA/TP 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 








QTQ AT A^mO 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


2000.0 


0.02 


0.79 




TCTQ AT A'^'^^ 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


2000.0 


0.02 


0.79 




OMP 


100.0 


100.0 


0.0 


0.0 


2131.6 


0.03 


0.78 


Hco 


S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


2000.0 


0.01 


0.79 










SNR = 5 












GTC AT AQQO 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 






O 


AT AQQO 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 






Uniol 
Sudd 


OA/TP 


1 nn n 


97.4 


0.0 


n n 

u.u 


139.6 






q OA/TP 

o- WiVlX 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


10.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


10.0 








^T^ AT A^t^D 

OlO ril^riOO W 


1 no n 


100.0 


0.0 


1 on n 

IUU.U 


2000.0 


0.04 


0.72 


-^^ s 

o ft 


T<5TC! AT A'^cin 
1 o 1 o - rl Ij r\ o o v-/ 


1 nn n 


100.0 


0.0 


1 nn n 

IUU.U 


2000.0 


0.04 


0.72 


c6 ft 


OMP 
S-OMl'-ALASSO 


100.0 
100.0 


1 nn n 
100.0 


n n 
0.0 


0.0 

100.0 


2131.6 
2000.0 


0.05 
0.03 


0.71 
0.72 










SNR = 1 












SIS-ALASSO 


100.0 


100.0 


0.0 


37.0 


11.1 






r! O 


ISIS-ALASSO 


100.0 


100.0 


0.0 


44.0 


10.8 






Unio] 
Supp 


OMP 


100.0 


97.4 


0.0 


0.0 


139.6 






S-OMP 


0.0 


100.0 


90.0 


0.0 


1.0 








S-OMP-ALASSO 


0.0 


100.0 


90.0 


0.0 


1.0 








SIS-ALASSO 


0.0 


100.0 


12.0 


0.0 


1761.3 


2.15 


0.37 


^ g 
O ft 


ISIS-ALASSO 


0.0 


100.0 


9.1 


0.0 


1819.3 


1.71 


0.38 


Hco 


OMP 
S-OMP-ALASSO 


99.0 
0.0 


100.0 
100.0 


0.0 
93.2 


0.0 
0.0 


2131.6 
136.0 


0.26 
11.65 


0.42 
0.03 
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Simulation 3: (n, 


p, s, T) = (100, 5000, 3, 150), Tnon- 


-zero — 80, p 


= 0.2 








A/l6tlioci iiani6 


M, C S 


Correct zeros Incorrect zeros 


M* = S 




l|B - Blli 

II 1 M 












SNR = 10 












AT AS^D 


1 nn n 

±uu.u 


100.0 


0.0 


100.0 


3.0 


~ 




n O 


mjC! AT A^cin 


1 nn n 


100.0 


0.0 


100.0 


3.0 






Unio] 
Supp 


OMP 


1 nn n 


99.8 


0.0 


0.0 


20.0 






o OA/TP 


1 nn n 


100.0 


0.0 


100.0 


3.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


3.0 








QTQ AT A^mO 


Qfi n 


100.0 


0.0 


96.0 


239.9 


0.02 


0.73 




TCTQ AT A'^'^^ 


1 nn n 


100.0 


0.0 


100.0 


240.0 


0.02 


0.73 




OMP 


100.0 


100.0 


0.0 


0.0 


257.1 


0.03 


0.72 


Hco 


S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


240.0 


0.01 


0.73 










SNR = 5 












GTC AT AQQO 


1 nn n 


100.0 


0.0 


100.0 


3.0 






O 


AT AQQO 


1 nn n 

lUU.U 


100.0 


0.0 


100.0 


3.0 






Uniol 
Sudd 


OA/TP 


1 nn n 


99.8 


0.0 


0.0 


19.6 






q OA/TP 

o- WiVlX 


1 nn n 

lUU.U 


100.0 


0.0 


100.0 


3.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


3.0 








^T^ AT A^t^D 


1 no n 


100.0 


0.0 


100.0 


240.0 


0.02 


0.72 


-^^ s 

o ft 


T<5TC! AT A'^cin 


1 nn n 


100.0 


0.0 


100.0 


240.0 


0.02 


0.72 


c6 ft 


OMP 
S-OMl'-ALASSO 


100.0 
iOO.O 


1 nn n 
100.0 


n n 
0.0 


0.0 

100.0 


256.6 
210.0 


0.03 
0.01 


0.72 
0.72 










SNR = 1 












SIS-ALASSO 


100.0 


100.0 


0.0 


92.0 


3.1 






r! O 


ISIS-ALASSO 


100.0 


100.0 


0.0 


94.0 


3.1 






Unio] 
Supp 


OMP 


100.0 


99.8 


0.0 


0.0 


20.3 






S-OMP 


100.0 


100.0 


0.0 


100.0 


3.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


3.0 








SIS-ALASSO 


99.0 


100.0 


0.0 


92.0 


240.1 


0.04 


0.70 


^ g 
O ft 


ISIS-ALASSO 


100.0 


100.0 


0.0 


94.0 


240.1 


0.03 


0.70 


Hco 


OMP 
S-OMP-ALASSO 


100.0 
100.0 


100.0 
100.0 


0.0 
0.0 


0.0 
100.0 


257.3 
240.0 


0.04 
0.02 


0.69 
0.70 
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Simulation 3: (n, 


p, s, T) = (100, 5000, 3, 150), Tnon- 


-zero — 80, p 


= 0.5 








A/l6tlioci iiani6 


M, C S 


Correct zeros Incorrect zeros 


M* = S 




l|B - Blli 

II 1 M 












SNR = 10 












AT AS^D 


1 nn n 

lUU.U 


100.0 


0.0 


98.0 


3.0 


~ 




n O 


mjC! AT A^cin 


1 nn n 


100.0 


0.0 


100.0 


3.0 






Unio] 
Supp 


OMP 


1 nn n 


99.8 


0.0 


0.0 


20.1 






o OA/TP 


1 nn n 


100.0 


0.0 


100.0 


3.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


3.0 








QTQ AT A^mO 


87 n 


100.0 


0.2 


85.0 


239.5 


0.08 


0.62 




TCTQ AT A'^'^^ 


88 n 


100.0 


0.1 


88.0 


239.8 


0.07 


0.62 




OMP 


100.0 


100.0 


0.0 


0.0 


257.1 


0.06 


0.62 


Hco 


S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


240.0 


0.03 


0.63 










SNR = 5 












GTC AT AQQO 


1 nn n 


100.0 


0.0 


97.0 


3.0 






O 


AT AQQO 


1 nn n 


100.0 


0.0 


96.0 


3.0 






Uniol 
Sudd 


OA/TP 


1 nn n 


99.8 


0.0 


0.0 


19.6 






q OA/TP 

o- WiVlX 


1 nn n 


100.0 


0.0 


100.0 


3.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


3.0 








^T^ AT A^t^D 

OlO ril^riOO W 


fin 


100.0 


0.2 


57.0 


239.5 


0.10 


0.61 


-^^ s 

o ft 


T<5TC! AT A'^cin 
1 o 1 o - rl Ij r\ o o v-/ 




100.0 


0.1 


80.0 


239.8 


0.08 


0.61 


c6 ft 


OMP 
S-OMl'-ALASSO 


100.0 
iOO.O 


1 nn n 
100.0 


u . u 
0.0 


0.0 

100.0 


256.6 
210.0 


0.06 
O.o:! 


0.61 
0.62 










SNR = 1 












SIS-ALASSO 


100.0 


100.0 


0.0 


56.0 


3.5 






r! O 


ISIS-ALASSO 


100.0 


100.0 


0.0 


70.0 


3.4 






Unio] 
Supp 


OMP 


100.0 


99.8 


0.0 


0.0 


19.9 






S-OMP 


100.0 


100.0 


0.0 


100.0 


3.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


3.0 








SIS-ALASSO 


1.0 


100.0 


2.3 


1.0 


235.1 


0.21 


0.58 


^ g 
O ft 


ISIS-ALASSO 


5.0 


100.0 


1.5 


3.0 


236.8 


0.16 


0.58 


Hco 


OMP 
S-OMP-ALASSO 


96.0 
67.0 


100.0 
100.0 


0.0 
0.2 


0.0 
67.0 


256.9 
239.5 


0.08 
0.05 


0.58 
0.59 
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Simulations: {n,p,s,T) = (100,5000,3,150), rnon_,cro = 80, p = 0.7 

Method name Ai* C S Correct zeros Incorrect zeros M,=S \S\ ||B-B||i 

SNR = 10 





SIS-ALASSO 


80.0 


100.0 


6.7 


80.0 


2.8 


n O 


ISIS-ALASSO 


85.0 


100.0 


5.0 


85.0 


2.9 


Unio] 
Supp 


OMP 


100.0 


99.8 


0.0 


0.0 


22.0 


S-OMP 


0.0 


100.0 


51.0 


0.0 


1.5 




S-OMP-ALASSO 


0.0 


100.0 


51.0 


0.0 


1.5 



SIS-ALASSO 


0.0 


100.0 


63.3 


0.0 


88.1 


3.93 


0.15 


ISIS-ALASSO 


0.0 


100.0 


61.0 


0.0 


93.6 


3.70 


0.16 


OMP 


0.0 


100.0 


12.0 


0.0 


230.2 


0.73 


0.28 


S-OMP-ALASSO 


0.0 


100.0 


57.6 


0.0 


101.8 


2.89 


0.19 



SNR = 5 



SIS-ALASSO 


79.0 


100.0 


7.0 


79.0 


2.8 






ISIS-ALASSO 


85.0 


100.0 


5.0 


83.0 


2.9 






OMP 


100.0 


99.8 


0.0 


0.0 


22.5 






S-OMP 


0.0 


100.0 


56.7 


0.0 


1.3 






S-OMP-ALASSO 


0.0 


100.0 


56.7 


0.0 


1.3 






SIS-ALASSO 


0.0 


100.0 


66.0 


0.0 


81.6 


4.15 


0.14 


ISIS-ALASSO 


0.0 


100.0 


64.2 


0.0 


85.9 


3.95 


0.15 


OMP 


0.0 


100.0 


16.5 


0.0 


219.8 


0.96 


0.26 


S-OMl'-ALASSO 


0.0 


100.0 


()i.2 


0.0 


|}:!.0 




0.18 



SNR = 1 





SIS-ALASSO 


89.0 


100.0 


3.7 


45.0 


3.5 


c. O 


ISIS-ALASSO 


92.0 


100.0 


2.7 


49.0 


3.5 


Uniol 
Supp 


OMP 


100.0 


99.8 


0.0 


0.0 


27.7 


S-OMP 


0.0 


100.0 


60.3 


0.0 


1.2 




S-OMP-ALASSO 


0.0 


100.0 


60.3 


0.0 


1.2 



SIS-ALASSO 


0.0 


100.0 


71.4 


0.0 


69.4 


4.76 


0.11 


ISIS-ALASSO 


0.0 


100.0 


68.9 


0.0 


75.3 


4.46 


0.12 


OMP 


0.0 


100.0 


29.3 


0.0 


196.8 


1.96 


0.23 


S-OMP-ALASSO 


0.0 


100.0 


64.6 


0.0 


85.0 


3.53 


0.16 
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Simulation 4: (n 


,p,s,T) = (150,4000,8,150), Tnon- 


-zero — 80, p 


= 0.2 








A/l6tlioci iiani6 


M, C S 


Correct zeros Incorrect zeros 


M* = S 




l|B - Blli 

II 1 M 












SNR = 10 












AT AS^D 


1 nn n 

lUU.U 


100.0 


0.0 


100.0 


8.0 


~ 




n O 


mjC! AT A^cin 


1 nn n 


100.0 


0.0 


97.0 


8.0 






Unio] 
Supp 


OMP 


1 nn n 


99.9 


0.0 


2.0 


11.7 






o OA/TP 


1 nn n 


100.0 


0.0 


100.0 


8.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


8.0 








QTQ AT A^mO 


'i'^ n 

oo.u 


100.0 


1.4 


35.0 


631.3 


0.55 


0.88 




TCTQ AT A'^'^^ 


1 nn n 


100.0 


0.0 


97.0 


640.0 


0.14 


0.89 




OMP 


100.0 


100.0 


0.0 


2.0 


643.7 


0.10 


0.89 


Hco 


S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


640.0 


0.09 


0.89 










SNR = 5 












GTC AT AQQO 


1 nn n 


100.0 


0.0 


85.0 


8.2 






O 


AT AQQO 


1 nn n 


100.0 


0.0 


78.0 


8.3 






Uniol 
Sudd 


OA/TP 


1 nn n 


99.9 


0.0 


2.0 


11.7 






q OA/TP 

o- WiVlX 


1 nn n 


100.0 


0.0 


100.0 


8.0 








S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


8.0 








^T^ AT A^t^D 

OlO r\.l-;r\.00 W 


9 


100.0 


4.5 


2.0 


611.7 


1.78 


0.77 


-^^ s 

o ft 


T<5TC! AT A'^cin 
1 o 1 o - rl Ij r\ o o v-/ 


7 n 
( .u 


100.0 


2.9 


6.0 


621.5 


1.29 


0.78 


c6 ft 


OMP 
S-OMl'-ALASSO 


100.0 

39.0 


1 nn n 
100.0 


U . U 

i.O 


2.0 
39.0 


643.7 
G33.8 


0.20 
0.18 


0.80 
0.80 










SNR = 1 












SIS-ALASSO 


0.0 


100.0 


90.5 


0.0 


0.8 






r! O 


ISIS-ALASSO 


0.0 


100.0 


87.6 


0.0 


1.0 






Unio] 
Supp 


OMP 


100.0 


99.8 


0.0 


0.0 


14.9 






S-OMP 


0.0 


100.0 


87.5 


0.0 


1.0 








S-OMP-ALASSO 


0.0 


100.0 


88.5 


0.0 


0.9 








SIS-ALASSO 


0.0 


100.0 


99.9 


0.0 


0.8 


29.62 


-0.01 


^ g 
O ft 


ISIS-ALASSO 


0.0 


100.0 


99.8 


0.0 


1.1 


29.61 


-0.01 


Hco 


OMP 
S-OMP-ALASSO 


0.0 
0.0 


100.0 
100.0 


31.1 
99.6 


0.0 
0.0 


447.7 
2.7 


10.11 
29.56 


0.32 
-0.00 



41 



Simulation 4: {n,p,s,T) = (150,4000,8,150), rnon-zcro = 80, p = 0.5 

Method name Ai* C S Correct zeros Incorrect zeros M,=S \S\ ||B-B||i 

SNR = 10 





SIS-ALASSO 


100.0 


100.0 


0.0 


80.0 


8.2 


n O 


ISIS-ALASSO 


100.0 


100.0 


0.0 


89.0 


8.1 


Unio] 
Supp 


OMP 


100.0 


99.9 


0.0 


2.0 


11.9 


S-OMP 


100.0 


100.0 


0.0 


100.0 


8.0 




S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


8.0 



SIS-ALASSO 


0.0 


100.0 


13.1 


0.0 


556.5 


4.24 


0.80 


ISIS-ALASSO 


80.0 


100.0 


0.2 


70.0 


638.9 


0.23 


0.89 


OMP 


100.0 


100.0 


0.0 


2.0 


643.9 


0.11 


0.89 


S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


640.0 


0.10 


0.89 



SNR = 5 



SIS-ALASSO 


100.0 


100.0 


0.0 


69.0 


8.4 






ISIS-ALASSO 


100.0 


100.0 


0.0 


47.0 


8.9 






OMP 


100.0 


99.9 


0.0 


2.0 


12.3 






S-OMP 


100.0 


100.0 


0.0 


100.0 


8.0 






S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


8.0 






SIS-ALASSO 


0.0 


100.0 


23.8 


0.0 


487.8 


7.53 


0.65 


ISIS-ALASSO 


0.0 


100.0 


7.6 


0.0 


592.5 


2.75 


0.75 


OMP 


99.0 


100.0 


0.0 


2.0 


644.4 


0.22 


0.80 


S-OMl'-ALASSO 


r.o 


100.0 


2.8 


7.0 


022.2 


1.01 


0.79 



SNR = 1 





SIS-ALASSO 


0.0 


100.0 


60.6 


0.0 


3.2 


c. O 


ISIS-ALASSO 


1.0 


100.0 


56.8 


1.0 


3.5 


Uniol 
Supp 


OMP 


100.0 


99.6 


0.0 


0.0 


23.5 


S-OMP 


0.0 


100.0 


87.5 


0.0 


1.0 




S-OMP-ALASSO 


0.0 


100.0 


87.5 


0.0 


1.0 



SIS-ALASSO 


0.0 


100.0 


99.3 


0.0 


4.7 


29.45 


-0.00 


ISIS-ALASSO 


0.0 


100.0 


99.2 


0.0 


5.1 


29.43 


-0.00 


OMP 


0.0 


100.0 


44.9 


0.0 


369.3 


15.05 


0.28 


S-OMP-ALASSO 


0.0 


100.0 


98.5 


0.0 


9.9 


29.39 


0.01 
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Simulation 5: {n,p,s,T) = (200, 10000,5,500), rnon-zoro = 400 

Method name M, C S Correct zeros Incorrect zeros M* = S \S\ ||B-B||i 

a = 1.5 





SIS-ALASSO 


53.0 


99.6 


9.4 


0.0 


41.1 


n O 


ISIS-ALASSO 


100.0 


99.8 


0.0 


0.0 


28.1 


Unio] 
Supp 


OMP 


100.0 


99.9 


0.0 


12.0 


10.0 


S-OMP 


100.0 


100.0 


0.0 


44.0 


5.6 




S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


5.0 



SIS-ALASSO 


0.0 


100.0 


68.9 


0.0 


936.0 


84.66 


0.66 


ISIS-ALASSO 


0.0 


100.0 


16.2 


0.0 


1791.9 


5.80 


0.96 


OMP 


100.0 


100.0 


0.0 


12.0 


2090.3 


0.06 


0.99 


S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


2000.0 


0.05 


0.99 



a = 2.5 



SIS-ALASSO 


53.0 


99.4 


9.4 


0.0 


61.4 






ISIS-ALASSO 


100.0 


99.3 


0.0 


0.0 


77.7 






OMP 


100.0 


99.9 


0.0 


10.0 


13.2 






S-OMP 


100.0 


100.0 


0.0 


44.0 


5.6 






S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


5.0 






SIS-ALASSO 


0.0 


100.0 


69.2 


0.0 


910.2 


85.82 


0.64 


ISIS-ALASSO 


0.0 


100.0 


17.5 


0.0 


1834.1 


7.23 


0.93 


OMP 


100.0 


100.0 


0.0 


10.0 


2093.3 


0.16 


0.96 


S-OMl'-ALASSO 


93.0 


100.0 


0.0 


93.0 


1999.9 


0.13 


0.90 



a = 4.5 





SIS-ALASSO 


40.0 


99.1 


12.0 


0.0 


92.5 


c. O 


ISIS-ALASSO 


100.0 


97.8 


0.0 


0.0 


226.8 


Uniol 
Supp 


OMP 


100.0 


99.8 


0.0 


1.0 


25.7 


S-OMP 


92.0 


100.0 


1.6 


46.0 


5.5 




S-OMP-ALASSO 


92.0 


100.0 


1.6 


92.0 


5.0 



SIS-ALASSO 


0.0 


100.0 


70.0 


0.0 


850.2 


88.65 


0.56 


ISIS-ALASSO 


0.0 


100.0 


27.4 


0.0 


1847.2 


15.79 


0.83 


OMP 


0.0 


100.0 


3.2 


0.0 


2040.9 


1.15 


0.88 


S-OMP-ALASSO 


0.0 


100.0 


10.2 


0.0 


1795.3 


2.38 


0.87 
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Simulation 5: {n,p,s,T) = (200, 10000,5,500), rnon-zero = 250 



Method name M, C S Correct zeros Incorrect zeros M* = S \S\ ||B-B||i 

a = 1.5 





SIS-ALASSO 


100.0 


99.7 


0.0 


0.0 


31.5 


n O 


ISIS-ALASSO 


100.0 


99.9 


0.0 


1.0 


14.3 


Unio] 
Supp 


OMP 


100.0 


99.7 


0.0 


0.0 


30.8 


S-OMP 


100.0 


100.0 


0.0 


20.0 


5.8 




S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


5.0 



SIS-ALASSO 


0.0 


100.0 


45.9 


0.0 


768.9 


25.98 


0.79 


ISIS-ALASSO 


0.0 


100.0 


5.3 


0.0 


1200.7 


1.00 


0.92 


OMP 


100.0 


100.0 


0.0 


0.0 


1287.6 


0.05 


0.92 


S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


1250.0 


0.03 


0.92 



a = 2.5 



SIS-ALASSO 


100.0 


99.6 


0.0 


0.0 


40.5 






ISIS-ALASSO 


100.0 


99.6 


0.0 


0.0 


44.3 






OMP 


100.0 


99.7 


0.0 


0.0 


32.0 






S-OMP 


100.0 


100.0 


0.0 


23.0 


5.8 






S-OMP-ALASSO 


100.0 


100.0 


0.0 


100.0 


5.0 






SIS-ALASSO 


0.0 


100.0 


46.2 


0.0 


757.5 


26.30 


0.74 


ISIS-ALASSO 


0.0 


100.0 


7.5 


0.0 


1205.2 


1.55 


0.86 


OMP 


100.0 


100.0 


0.0 


0.0 


1288.6 


0.14 


0.87 


S-OMl'-ALASSO 


92.0 


100.0 


0.0 


92.0 


1219.9 


0.08 


0.87 



a = 4.5 





SIS-ALASSO 


98.0 


99.6 


0.4 


0.0 


48.0 


c. O 


ISIS-ALASSO 


100.0 


99.0 


0.0 


0.0 


104.0 


Uniol 
Supp 


OMP 


100.0 


99.7 


0.0 


0.0 


36.1 


S-OMP 


1.0 


100.0 


19.8 


1.0 


4.7 




S-OMP-ALASSO 


1.0 


100.0 


19.8 


1.0 


4.2 



SIS-ALASSO 


0.0 


100.0 


48.4 


0.0 


713.1 


27.64 


0.62 


ISIS-ALASSO 


0.0 


100.0 


22.8 


0.0 


1080.7 


5.57 


0.71 


OMP 


0.0 


100.0 


2.3 


0.0 


1264.0 


0.70 


0.75 


S-OMP-ALASSO 


0.0 


100.0 


19.9 


0.0 


1002.0 


2.26 


0.73 
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Simulation 5: {n,p, s, T) = (200, 10000, 5, 500), Tnon-zero = 100 

Method name M., C S Correct zeros Incorrect zeros M,=S \S\ ||B-B||i 

a = 1.5 





SIS-ALASSO 


100.0 


99.9 


0.0 


1.0 


10.9 


n O 


ISIS-ALASSO 


100.0 


100.0 


0.0 


56.0 


5.7 


Unio] 
Supp 


OMP 


100.0 


98.0 


0.0 


0.0 


205.8 


S-OMP 


99.0 


100.0 


0.2 


4.0 


6.0 




S-OMP-ALASSO 


99.0 


100.0 


0.2 


99.0 


5.0 



SIS-ALASSO 


0.0 


100.0 


19.4 


0.0 


411.0 


2.86 


0.60 


ISIS-ALASSO 


17.0 


100.0 


0.5 


16.0 


498.0 


0.06 


0.62 


OMP 


100.0 


100.0 


0.0 


0.0 


726.4 


0.19 


0.60 


S-OMP-ALASSO 


99.0 


100.0 


0.2 


99.0 


499.0 


0.02 


0.62 



a = 2.5 



SIS-ALASSO 


100.0 


99.9 


0.0 


1.0 


11.0 






ISIS-ALASSO 


100.0 


99.9 


0.0 


0.0 


12.4 






OMP 


100.0 


98.0 


0.0 


0.0 


205.8 






S-OMP 


0.0 


100.0 


20.0 


0.0 


4.9 






S-OMP-ALASSO 


0.0 


100.0 


20.0 


0.0 


4.0 






SIS-ALASSO 


0.0 


100.0 


19.6 


0.0 


408.8 


2.92 


0.54 


ISIS-ALASSO 


0.0 


100.0 


2.5 


0.0 


495.2 


0.21 


0.56 


OMP 


100.0 


100.0 


0.0 


0.0 


726.4 


0.54 


0.53 


S-OMl'-ALASSO 


0.0 


100.0 


20.0 


0.0 


100.0 


0.83 


0.52 



a = 4.5 





SIS-ALASSO 


98.0 


100.0 


0.4 


1.0 


9.8 


c. O 


ISIS-ALASSO 


97.0 


99.9 


0.6 


0.0 


17.4 


Uniol 
Supp 


OMP 


100.0 


98.0 


0.0 


0.0 


206.4 


S-OMP 


0.0 


100.0 


41.2 


0.0 


3.6 




S-OMP-ALASSO 


0.0 


100.0 


41.2 


0.0 


3.4 



SIS-ALASSO 


0.0 


100.0 


27.6 


0.0 


367.3 


3.48 


0.41 


ISIS-ALASSO 


0.0 


100.0 


19.9 


0.0 


413.1 


1.33 


0.42 


OMP 


4.0 


100.0 


1.4 


0.0 


720.0 


1.79 


0.41 


S-OMP-ALASSO 


0.0 


100.0 


41.2 


0.0 


295.9 


4.66 


0.35 
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